Tool for labelling training images for a vehicle characteristic detection model

ABSTRACT

A tool used in an image processing system assists a user to train one or more statistical image models or classification engines (e.g., the CNN models) that are used to detect damaged areas on a vehicle, to detect damage types and/or to detect segments of the depiction of the vehicle. The tool enables a user to select and annotate various different training images to be used to train the models, wherein each of the training images depicts damage of one or more damage types to various different vehicles or automobiles (including automobiles of different years/makes/models). The tool displays each of a set of selected training images and enables a user to indicate, on the displayed selected training image, using an electronic pen, a touch screen or any other type of selector device, one or more sets of pixels within the displayed image that are associated with or that depict damage to the vehicle, one or more sets of pixels within the displayed image that are associated with or that depict a particular type of vehicle damage and/or one or more sets of pixels within the displayed image that are associated with or included in a particular segment of the depiction of the vehicle within the image. The marked or tagged images are then used to train an image model to detect vehicle damage, types of vehicle damage and/or vehicle segments in new vehicle images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 63/186,717 filed on May 10, 2021 and entitled“IMAGE PROCESSING SYTSEM AND METHOD FOR DETECING PRECISE LOCATIONS,SIZES AND TYPES OF DAMAGE TO AN OBJECT,” the disclosure of which ishereby incorporated by reference in its entirety.

TECHNICAL FIELD

This patent relates to an image processing system and technique, andmore particularly, to an image processing system and methodology thatdetects highly precise locations, sizes and/or types of changes to anobject.

DESCRIPTION OF THE RELATED ART

Image processing systems typically operate on images, such as photos,digital pictures, digital video frames, computer generated images, etc.,to enhance the image in some manner or to detect certain features orcharacteristics of an image, such as to determine information aboutobjects in the image, to recognize persons or things in the image, etc.For example, there are many image processing systems that performcharacter or facial recognition in images to identify text, people,particular buildings, or other features of objects within images inorder to automatically identify people, objects, or other featuresdepicted within the image. In many cases, these image processing systemsuse statistical processing techniques to detect particular features orcharacteristics of the set of pixels that make up a feature based on thesimilarity of the characteristics of the image pixels to other images ofthe same or similar object, feature, etc. being detected. In othercases, image processing systems look for and detect defects in the imagecaused by the camera, such as red eye detection, distortion detection,color balance detection, etc., all with the goal of correcting oraltering the image to make the image a better or more realistic or morepleasing image. As further examples, some image processing systemsoperate to perform edge detection to detect objects within an image, tofilter images in various manners to reduce or enhance particularfeatures of the image, etc. The goal of most of these image processingsystems is to create a better or more useful image, or to detectfeatures within the image for some other purpose.

Traditionally, there have been very few image processing systems thatcan quickly and effectively detect changes that have occurred to anobject as depicted in an image to thereby detect the manner in which anobject depicted in the image has changed from a known state or conditionto, for example, a degraded or an upgraded state or condition. Forexample, known image processing systems are unable to quickly andeffectively detect, isolate or quantify damage that may have occurred toan automobile in an accident, changes to buildings or other structuresthat may have occurred due to further construction on the buildings, dueto tornado or flood damage, etc. While it is possible to compare imagesof the same object at different times to detect changes to the image ofthe object, it is difficult to automatically detect or quantify actualchanges to the object based on this comparison of the images for anumber of reasons. In particular, such a comparison requires the imagesof the object to be from the same perspective, angle, distance, etc.which is difficult to achieve in practice. Moreover, a high fidelitycomparison would normally require that the images be obtained from thesame camera to account for distortions that would typically beintroduced by different cameras taking images at different times. Stillfurther, a simple comparison of images of the same object from the sameperspective and using the same camera may still result in the detectionof changes that are not substantive in nature due to differences inlighting, surrounding objects, etc. Moreover, it is difficult toquantify the nature or type of changes even if such changes are bedetected.

More recently, there have been some advances in using image processingtechniques to analyze one or more images of an object to detect, moreprecisely, areas of change, e.g., damage on the object. For example,U.S. Pat. No. 9,886,771 describes an image processing system thatcreates and uses statistically based models of known automobiles todetect damage to an automobile caused by, for example, an accident. Theoutput of this model includes a “heat map” illustrating particular areasof damage detected in various locations or panels of a vehicle ascompared to a non-damaged vehicle of the same make, model and year. Thisheat map can be used by a viewer to grossly determine where damage islikely to exist on a particular vehicle based on images or photos of thevehicle and the viewer can use the heat map as part of a process toestimate repair work that needs to be performed on the vehicle andthereby to estimate repair costs associated with repairing the vehicle.However, unfortunately, a user such as an estimator must still analyzethe images along with the heat map to estimate the size of the damage(such how much of a particular panel is damaged, the size of the damageas compared to the panel size, etc.) as well as the type of damage (forexample, is the damage a scratch, a dent, a tear, a misalignment, etc.)in order to estimate the repairs that need to be performed and the costsof such repairs. Still further, the techniques described in this patentare computationally complex and require the creation of a number ofartificial intelligence (AI) models for different automobiles anddifferent parts of different automobiles.

SUMMARY

An image processing system includes various different processingcomponents including one or more classification engines, such asstatistical image models or convolutional neural network (CNN) models,and analytic routines that process various images of an object, such asimages of an automobile or other vehicle, to detect and quantify changesto or on the object, such as damage caused to an automobile in anaccident. More particularly, the image processing system may obtain orbe provided with a set of target images of the object, with the targetimages generally depicting different views or areas of the changedobject. In one example, the target images may depict different views ofthe exterior of a vehicle, such as a front planar view, corner orperspective views, side planar views, and a rear planar view of anautomobile. A single target image may depict only a single planar viewof the vehicle, or a single target image may depict multiple planarviews of the vehicle (that is, may depict a corner or perspective viewof the vehicle). The image processing system may then process thevarious target images of the object, such as any of a set of corner,side, front, back and interior images of an automobile, to determinechange characteristics of the object, such as the precise location of,the relative size of and the type of damage to an automobile depicted inthe images.

In one example, the image processing system may receive or obtain a setof target images of a vehicle, as one or more corner, side, front, backand/or interior images of an automobile or vehicle. In one case, theimage processing system may analyze one or more of the target imagesusing a model-based classification engine approach to determine moreprecise object identification information describing or indicating thetype of the object within the images, such as an indication of the year,make and model, and in some cases, the level of trim (abbreviated hereinas “Y/M/M”) of an automobile or vehicle depicted within the set oftarget images. In another case, the image processing system may receivethis identification information from a user or a file. The imageprocessing system may also or instead analyze each of the target imagesto determine the view of the object that is depicted in each of thetarget images and tag each of the target images with the type of view(e.g., front, right side, left side, back, or perspective) of the objectand/or with the zoom level of the object as depicted within the targetimage. More particularly, the image processing system may implement oneor more statistical or model-based analyses on some or all of theprovided target images, such as on each of a set of target imagesdepicting one or more of a corner, a side, a front and a back of avehicle, including images at various zoom levels, to determine thenature or view of the image, including the location of the image withrespect to the object (e.g., the front right corner view, the left rearcorner view, the passenger side view, the driver side view, the frontview, the back view, etc.), as well as a zoom level of the view of theobject (no zoom, zoomed, or highly zoomed, for example). Upondetermining the view and/or the zoom level of the object within theimage, the image processing system may tag or label each target imagewith any or all of the Y/M/M, the view, and the zoom information tocreate a set of tagged target images.

The image processing system may then select, from the tagged targetimages, a subset of tagged target images that depict the entire objector a set of desired views of the object, such as eight standard views ofan automobile (front, right front corner, passenger side, right rearcorner, rear or back, left rear corner, driver side, and left frontcorner) at appropriate zoom levels (no zoom, medium zoom, large zoom).Moreover, the image processing system may cull or reduce the set oftagged target images to be used in further processing by eliminatingimages that are too zoomed in, too far away, have glare or other visualor camera artifacts, etc. that prevent the target image from being ableto be processed correctly in later steps, to eliminate duplicate viewsof the object, etc. This reduced set of images is referred to herein asa selected set of tagged target images. Of course, the image processingsystem may perform the culling process prior to the tagging process ifdesired. In addition, the image processing system may perform imageenhancement to the tagged target images, such as to perform glarereduction, color correction, etc., ultimately to create a set ofselected tagged target images that depict or show the object fromdifferent perspectives (and possibly zoom levels) and which can be usedsingly or as a group in later steps of the image processing techniquesdescribed herein to identify more precisely a set of changecharacteristics of the object as depicted in the selected tagged targetimages. In one example, such change characteristics may be damagecharacteristics identifying particular aspects of damage to the object.In a further example, such damage characteristics may be a type ofdamage, a precise location of the damage on the object (e.g., relativeto some known component or segment of the object), and/or a precise sizeof the damage to the object.

More particularly, the image processing system may analyze each of theselected tagged target images to determine if there is any damage to theobject depicted in each or any of the selected tagged target images. Inone embodiment, the image processing system may implement one or more ofthe techniques described in U.S. Pat. No. 9,886,771 to obtain a heat mapcorresponding to each of the selected tagged target images, with each ofthe heat maps identifying, at a pixel level, where there is damage orlikelihood of damage on the object, as depicted in the selected taggedtarget image. In another case, the image processing system may use acharacterization engine and, more particularly, may use a CNN basedimage model to process each of the pixels of each of the selected taggedtarget images to determine the particular pixels of the image (or of theobject depicted within the image) that depict the presence of damage tothe object, and the likelihood of the pixels depicting damage. In thiscase, the characterization engine or CNN model or transformer model maybe developed or trained using a training engine that analyzes aplurality of images of objects (e.g., different automobiles) damaged invarious different manners which have been annotated, on a pixel by pixelbasis, to indicate which pixels of each image represent damaged areas ofthe objects and which have also been annotated, on an image basis, toindicate the view and/or zoom level of the image.

Moreover, in one example, the image processing system may furtheranalyze each of the selected tagged target images in which damage isdetected to determine a type of damage to the object depicted in theselected tagged target image. Here, the image processing system maystore and implement a damage type detection model, such as a statisticalimage model, which has been trained on various training images todetermine or categorize the type of damage depicted in an image asfalling into one or more of a preset number of damage types (or as notbeing determinable). In some cases, the image processing system mayperform a model-based classification or statistical analysis on each ofthe selected tagged target images (such as the images for which therespective heat map indicates the likely existence of damage) todetermine the type of damage depicted within the image and moreparticularly the type of damage depicted at each of the pixels of theimage which were identified as depicting the presence of damage to theobject by the heat map of that image. In one example, in which images ofan automobile are analyzed, the image processing system may detect ordetermine if the damage is associated with one of various preset typesof damage, including, for example, a dent, a scratch, a tear, amisalignment, a hole, a missing part, a kink, a twist, etc. Moreover,the image processing system may, in some cases, determine that damage ina particular image falls into multiple categories, as there may bevarious different parts of the detected damage (as identified by theheat map for example) that fall into different damage types orcategories. Still further, the image processing system may determine aprobability of the type of damage for each pixel such that a singlepixel may have various probabilities of depicting different types ofdamage or may even depict two or more types of damage. In one example,the image processing system may use a characterization engine and, moreparticularly, may use a CNN-based image model to process each of thepixels of each of the selected tagged target images to determine thedamage type or damage types depicted by each pixel of the image (or ofthe object depicted within the image) and, if desired, a probabilitythat the pixel depicts each of those types of damage. In this case, thecharacterization engine or CNN model or transformer model may bedeveloped or trained using a training engine that uses a plurality ofimages of objects (various different automobiles of different Y/M/M forexample) damaged in various different manners which have been annotated,on a pixel by pixel basis, to indicate which pixels of the imagerepresent damaged areas of the objects and further to indicate the typeof damage associated with each such damaged pixel, and which have alsobeen annotated, on an image basis, to indicate the view and/or zoomlevel of the image.

Still further, the image processing system may analyze each of theselected tagged target images of the object to perform a segmentation ofthe depiction of the object within the selected tagged target images tothereby create a set of segmented tagged target images in which theboundaries of each of a set of known segments or components depicted inthe image are identified. Different segments of the depiction of anobject typically are mutually exclusive areas within the image (such asdifferent side or planar views depicted within a single perspectiveimage, different portions of one of the planar views depicted within asingle planar or perspective image, etc.). The different segmentsdepicted within a segmented tagged target image may be indicated ordifferentiated by respective borders, respective colors, labels, and/orother suitable indicators. In one case, each pixel of the segmentedtagged target images (also called segmented target images, segment maps,or segment masks) may be labeled with a respective segment in which thepixel is included, and may also, if desired, be labeled or annotatedwith a probability that the pixel is included in the labeled segment. Insome cases, any particular pixel may be labeled as being potentiallyassociated with (e.g., potentially included in) multiple differentsegments and a respective probability of being included in each segment,or may be labeled as being associated with or being included in nosegment at all. Components or segments can be any desired knownsubdivision of the depiction of the object, such as one or more planarviews of the object (e.g., front view, rear or back view, right sideview, left side view, top or birds-eye view, etc.), parts of a planarview of an object (e.g., automobile body panels such as a hood, a door,a trunk, a front grill, etc.), and the like. In one example, the imageprocessing system may use a characterization engine and, moreparticularly, may use a CNN-based image model to process each of thepixels of each of the selected tagged target images to determine thesegment name or segment type in which that pixel is included and, ifdesired, a probability that the pixel is included in that segment. Inthis case, the characterization engine (or CNN model or transform) maybe developed or trained using a training engine that analyzes aplurality of images of objects (various different automobiles of thesame or different Y/M/M) damaged in various different manners which havebeen annotated, on a pixel by pixel basis, to indicate which pixels ofthe image are included in which segments of the depiction of the object,and which have also been annotated, on an image basis, to indicate theview and/or zoom level of the image.

In one case, the image processing system may include a tool that helps auser train each of the statistical image models or classificationengines (e.g., the CNN models) that are used to detect damaged areas ona vehicle, to detect damage types, and to detect segments of thedepiction of the vehicle by enabling a user to select and annotatevarious different training images to be used to train the models,wherein each of the training images depicts damage of one or more damagetypes to various different vehicles or automobiles (includingautomobiles of different Y/M/M). Generally, the tool may display each ofthe selected training images and enable a user to indicate, on thedisplayed selected training image, using an electronic pen, a touchscreen or any other type of selector device, one or more of (1) a set ofpixels within the displayed image that are associated with or thatdepict damage to the vehicle, (2) a set of pixels within the displayedimage that are associated with or that depict a particular type ofdamage, and/or (3) a set of pixels within the displayed image that areassociated with or included in a particular segment of the depiction ofthe vehicle within the image (e.g., associated with a particular planarview of the vehicle, with a particular automobile panel, etc.). The usermay, for example, be able to draw a line around the pixels associatedwith damage in general, pixels associated with a particular type ofdamage, such as a dent or a scratch depicted in the displayed trainingimage, or associated with a particular vehicle body panel within theimage, to indicate the pixels in the training image that depict damagein general, that depict damage of a particular type and/or that areincluded in particular segments of the depiction of the vehicle. Thetool may then enable the user to associate or label the selected pixelsof the displayed training image as being damaged or illustrating damage,as being associated with a particular type of damage, such as with adent, a scratch, a hole, a tear, etc., and/or as being associated with aparticular segment of the depiction of the object, such as with aparticular body panel of a vehicle. As an example, the model building ortraining tool may enable the user to label or associate a particulartype of damage or a particular segment of the depiction of an objectwithin the image with the selected pixels using a drop down menu, andthe system may then mark the training image and, in particular, theselected individual pixels of the training image with a particulardamage type or a particular segment as identified by the user or modeltrainer. In some cases, the user may also be able to indicate the viewof the image (e.g., right front corner view, side view, back view, etc.)and/or an approximate zoom level of the image (no zoom, moderate zoom,high zoom, etc.). These preprocessed images, referred to herein asdamage training images, damage-typed training images and segmentedtraining images, respectively, can then be advantageously used to trainstatistical or classification models (in one example, CNN models) thatare then used to process pixels in new images (e.g., target images) (1)to detect a probability of damage at respective locations, on thevehicle, depicted by each of those pixels, (2) to detect a type ofdamage and/or a probability of a particular type of damage at respectivelocations depicted by each of those pixels and/or (3) to detect therespective segment of the depiction of the object (and/or a probabilityof the respective segment of the depiction of the object) in which eachof those pixels is included.

Once a segmented target image or segmentation map for a target image isdetermined, the image processing system may overlay or otherwise comparethe segmentation map or segmented target image for a particular taggedtarget image with the particular tagged target image and label eachpixel in the selected tagged target image being processed with one ormore segment identifiers (so that each pixel depicted in the selectedtagged target image is identified as being included in corresponding oneor more segments of the depiction of the object). The image processingsystem may also or instead determine the boundaries of each of a numberof segments of the object as depicted in a target image being processedwith respect to the damage areas of the object by overlaying thesegmentation map or segmented target image for a particular selectedtagged target image with the selected heat map for the selected taggedtarget image being processed or with the damage-typed image for theselected tagged target image being processed. In other cases, such aswhen the selected tagged target image being processed is a corner view,the image processing system may perform a warping technique on theselected tagged target image and/or on its corresponding segmentationmap and/or heat map and/or damage-typed image to flatten the image(e.g., to make a two-dimensional view of the object as depicted in theimage to account for or compensate for three dimensional aspects of theobject depicted in the image). This warping technique may adjust for theangle of the camera used when taking the target image to make thevarious different parts of the object depicted within the image warpedto the same view or perspective and/or to make a three dimensionalobject image into a two dimensional image to keep all parts of theobject at the same approximate scale (and to eliminate three-dimensionperspective attributes of the image). In addition, the image processingsystem may warp each of the segmented target images using the samewarping technique, if necessary, and may warp the heat maps or heatmasks, or the damage-typed images for the tagged target images beingprocessed in the same manner.

In one case, the image processing system may overlay a heat map of atagged target image (depicting areas of damage within the image) withthe segmented target image (which has each pixel thereof labeled with asegment of the object with which it is associated or in which it isincluded) and/or a damaged-typed tagged target image (which has eachpixel corresponding to damage within the object depicted therein labeledwith a damage type) to determine the pixels within the tagged targetimage that are damaged or that are likely damaged (or that make up adepiction of a damaged portion of the object within the tagged targetimage). The image processing system may use this overlapping or overlayto determine damage or change characteristics of the object within theimage in a highly precise manner. In particular, the image processingsystem may compare the damage as depicted in a target image (based onthe labeled pixels of the corresponding heat map or damage-typed image)to a corresponding segment outline or description of the object (asindicated by the corresponding segmented target image) to therebyprecisely quantify the location and size of the damage or other changeto the object with respect to the segment in which the depiction of thedamage exists. In some cases, the image processing system may determinean area of each segment within a target image that depicts damage, therelative size of the damage depicted within the segment, such as apercentage of the segment in which damage is depicted, the size of thedamage in terms of the length and width of the damage as compared to thelength and width of the segment, the location of the damage as depictedwithin in the segment, etc.

Still further, in some cases, the image processing system may determinethat the depiction of a particular area of damage to the object spansmultiple ones of the selected tagged target images being processed(e.g., that a particular damaged area of the object is not depictedfully in a single tagged target image because different parts ofparticular damaged area are depicted in different ones of the taggedtarget images being processed). To determine the change characteristicsin this case, the image processing system may stitch various ones of theprocessed images together (e.g., the selected tagged target images, orthe segmented target images, or the damage-typed target images) and/orstitch various segments depicted within the processed images together tocreate a stitched image (e.g., a single composite or panoramic image)that illustrates the complete, particular damaged area. The changecharacteristics may then be determined based on a comparison of thestitched image and a base segmentation image or map depicting the entireparticular damaged area of the object. In some cases, the images orsegments thereof being processed may be stitched together such thatimages of a higher zoom level are stitched within or with images of alower zoom level, which provide better resolution for determining thechange characteristics in various different parts of the stitched image.In still other cases, the images or segments being processed may bestitched together to illustrate a damage site that is depicted acrossmultiple segments, to enable the damage site to be characterized andquantified as a whole, even though the damage site is depicted acrossmultiple segments.

In any event, after all of the images have been processed in some or allof the manners described above, the image processing system maydetermine a set of segments of the depiction of the object in whichchanges (e.g., damage) are depicted. For example, the image processingsystem may determine the segments (e.g., planar views, depictions ofpanels of an automobile) that depict complete or partial damage sitestherein. (In some cases, the depiction of damage sites may span multiplesegments.) The image processing system may also determine or quantifythe size and/or location of each damage site with respect to thesegment(s) over which the depiction of a particular damage site spans bydetermining or calculating the number of pixels included in thedepiction of the damage site as compared to the number of pixelsincluded in one or more of the segments over which the depiction of thedamage site spans, or the number of pixels along the length and/or widthand/or height of the depicted damage site as compared to the number ofpixels along the length and/or width and/or height of one or moresegments over which the damage site spans. Moreover, the imageprocessing system may measure or determine an actual size of the damagesite based on one or more comparisons of the damage site pixels to acorresponding base segment map in which the damage site exists, and theknown size of the base segment within the base segment map. Stillfurther, the image processing system may determine a location of adamage site with respect to one or more segments over which thedepiction of the damage site spans (such as in a particular quadrant oron a grid associated with one or more segments, and/or with respect toone or more waypoints or features corresponding to one or moresegments).

Moreover, after analyzing each of the selected target images byperforming a segmentation, damage detection, characterization and/orsizing with respect to each of the segments depicted within each of theselected tagged target images, the image processing system may determinea complete list of damage (as some of the same damage sites may bedepicted in multiple ones of the processed images) as well as a moreprecise description of the damage (or other change characteristics)including, for example, the damage location (e.g., the damage locationdepicted on each segment), the damage size (as compared to the segmentsize and/or a physical measurement of the segment or object) and/or adamage type for each of the damage sites. This list then quantifies, ina very precise manner, the location, size and/or type of damage depictedon each segment (e.g., each vehicle panel) which can then be used toquantify the complexity of the damage or the complexity of the estimateto be performed. Moreover, if desired, this damage information may bepresented graphically to a user in any desired manner, such as by beingprovided to the user in a list, on one or more two-dimensional orthree-dimensional annotated segment maps for a vehicle with the damageareas depicted in each segment indicated (e.g., using a color), with thetype of damage at each site indicated (e.g., with a color or a label orother identifier) and/or with the size or other characteristic of thedamage indicated (e.g. with text or a scale). This list or depiction ofdamage may be used to route the claim to more or less experiencedadjustors or estimators so as to reduce the overall estimation costs byonly using more experienced estimators (which are typically moreexpensive) on more complex claims and less experienced estimators onmore simple claims. Still further, the output of the analysis may beused to perform additional steps associated with automatic orsemi-automatic repair and cost estimation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary image processing system that can be used todetect changes in objects, such as to detect damage to automobiles,buildings, and the like, and to quantify the changes, such as bydetermining one or more change characteristics of the object, includinga type of change to the object and/or a precise location and/or a sizeof the change on the object.

FIG. 2 depicts a block diagram of an exemplary image processing systemand techniques that may be used to detect change characteristics of anobject, such as types, locations, and sizes of damage on the object.

FIGS. 3A-3G depict various images of an automobile associated withdifferent image processing steps performed to detect one or more changecharacteristics on an object depicted in the images.

FIGS. 4A-4C depict block diagrams of various different CNN model-baseddetectors that each implement a CNN to detect damage to a vehicle withinan image, to detect a damage type of damage to a vehicle within animage, and/or to detect one or more segments of a vehicle within animage.

FIG. 4D depicts a block diagram of a training module that may be used toproduce or create the CNNs used in the models of FIGS. 4A-C from variousdifferent types of training images.

FIG. 5A depicts a training tool that may be used to annotate trainingimages for use in the training module of FIG. 4D to create the CNNs usedin the models of FIGS. 4A-4C.

FIG. 5B depicts a flowchart of an annotation routine that may beimplemented by the training tool of FIG. 5A.

FIGS. 6A-6D depict display screens associated with or created by themodel training tool of FIGS. 5A and 5B that illustrate different aspectsof the training tool.

FIG. 7 depicts a block diagram of an example damage detailer routine,including an image warping routine and an image stitching routine, usedto process images to determine a precise size and location of damage toan object.

FIG. 8A depicts a block diagram of an example warping module which maybe included in the image processing system of FIG. 1 or FIG. 2.

FIG. 8B depicts a flow diagram of an example method for warping animage.

FIG. 9A depicts a block diagram of an example stitching module which maybe included in the image processing system of FIG. 1 or FIG. 2.

FIG. 9B depicts a flow diagram of an example method for stitchingtogether two images to form an integral image.

FIGS. 10A-10C depict example display screens illustrating detaileddamage on an automobile as determined by the image processing system 100and/or by the image processing system of FIG. 2.

DETAILED DESCRIPTION

FIG. 1 illustrates one example of an image processing system 100 whichmay be used to determine changes to a target object and to determinetherefrom one or more change characteristics of the object including,for example, a type of change, a location of the change on the object, asize of the change on the object, or any other precise description ofthe change(s) to the object. In the examples provided herein, the imageprocessing system 100 will be described as being used to detect changein the form of damage to an object and the object will be described as avehicle and, in particular, an automobile. However, it should be notedthat the image processing system 100 can be used to detect other typesof changes (including for example upgrades, additions, repairs,degradation, etc.) to any type of object including, for examplebuildings, land, bridges, roads, forests, jewelry, etc. Moreover, theterms “vehicle” and “automobile” are used herein to include any type ofvehicle, including powered and non-powered and motorized andnon-motorized vehicles.

Generally speaking, the image processing system 100 operates on a numberof target images of a “changed,” e.g., damaged, object to determinedifferences between the object as depicted in the one or more of thetarget images of the object and one or more images or depictions of thepre-changed object, or between the depicted object and a base model ofthe object representing the object prior to the changes occurring to theobject. Concurrent with or after changes to the target object have beendetermined, the image processing system 100 determines or quantifies thechanges in one or more precise manners, such as by determining a type ofeach change to the object, a precise location of each change to theobject, and/or a precise measurement and/or size of each change withinor on the object. These change characteristics may then be used todetermine secondary characteristics or features associated with thetarget object, such as how to process the target object in a moreefficient manner, costs associated with repairing or replacing thetarget object or parts thereof, the time it may take to repair thetarget object, the progress the target object has undergone in changingstates, etc.

As illustrated in FIG. 1, the image processing system 100 may includevarious user interface devices 102, which may be general purposecomputers 102, handheld devices 102A, workstations, etc., connectedthrough one or more communication networks 104 to one or more furthercomputing devices 106, which may operate as servers and/or databases ina traditional client-server or cloud-based network. For example, thefurther computing devices or servers 106 may include one or moredatabases 108, 109 and 110, and one or more servers 112 and 114 havingcomputer processing and storage capabilities.

In this case, the database 108 may store various processing componentsor image files that are used to train and implement one or morestatistical models to perform various image processing steps describedherein. In particular, the database 108 may store one or more baseobject models 120 (which may be, for example, two-dimensional and/orthree-dimensional object models) defining or depicting one or more baseobjects upon which change detection image processing may be performed.In one example, the base object models may be two or three-dimensionalrenditions, maps, depictions or other digital representations of variousdifferent automobiles (e.g., of automobiles of different years, makes,models and/or trim-types) in their original or pristine condition (i.e.,prior to any damage thereto). Additionally, the database 108 may storebase segment or base segmentation models or files 121 that depict,describe or define various segments or components of the one or morebase objects within the base object models 120. In one example, in whichthe image processing system 100 may be used to detect changes toautomobiles, such as to detect damage caused to an automobile in anaccident, the base object models 120 may be two-dimensional orthree-dimensional models of various automobiles based upon which changesmay be detected. Generally in this case, a different base object model120 will be stored for each year/make/model or year/make/model/trim-typeof automobile that is to be analyzed for damage. Moreover, a differentbase segment model or file 121 can be stored for each year/make/model oryear/make/model/trim-type of automobile that is to be analyzed fordamage. In this case, the base segment model 121 may identify varioussegments or components of the base object, such as body panels of anundamaged automobile (e.g., a hood, doors, trunk, grille, etc.),respective measurements thereof, respective locations thereof on theautomobile, etc. Of course, the base segment models 121 may be part ofthe base object models 120. Base object models 120 and segment objectmodels 121 may include or otherwise indicate physical measurementsand/or dimensions of respective vehicles, such as overall dimensions,respective dimensions of each segment (e.g., length, width, height,length of principal axis, length of shortest axis, etc.), and the like.

Moreover, the database 108 may store training images that can be used totrain one or more statistical models as described in more detail herein,including for example damage detection training image files 122,damage-typed training image files 123 and segmented training image files124. The damage detection training image files 122 may be files thatinclude images of changed (e.g. damaged) base objects illustratingvarious manners in which changes (damages) to the base objects withinthe base object model 120 may occur or have occurred in the past. Insome cases, images of non-changed (non-damaged) base objects may beincluded in the training image files 122 but this is not typicallynecessary. In some cases, these damage detection training image files122 can be images of different automobiles (of various different Y/M/Ms)that have been involved in different accidents and may also includeimages of non-damaged automobiles of different Y/M/Ms. Likewise, each ofthe damage detection training image files 122 may include an informationfile 122A associated therewith that provides information about theobject within the associated damage detection training image 122,including, for example, the Y/M/M of the vehicle depicted in the image122, an indication as to whether or not damage is depicted orillustrated in the image 122, the view or perspective of the objectwithin the image 122 (e.g., a side view, a right front corner view,etc.), the approximate or gross zoom level of the image 122 (e.g., nozoom, moderate zoom, large zoom), etc. Preferably, the training images122 are annotated on a pixel by pixel basis so that each pixel of theimage 122 is annotated in the file 122A with an indication of whetherthe pixel does nor does depict damage to the vehicle. Likewise, thedamage-typed training image files 123 depict various ones of the baseobjects within the base object model (e.g., automobiles of the same ordifferent Y/M/Ms) with various different types of damage thereon, suchas scratches, dents, misalignments, tears, holes, etc. In this case,each of the damage-typed training image files 123 includes aninformation file 123A associated therewith that provides informationabout the type of damage to the object within the associated trainingimage 123, including, for example, the Y/M/M of the vehicle depicted inthe image 123, an indication as to whether or not damage is depicted orillustrated in the image 123, the view or perspective of the image 123(e.g., a side view, a right front corner view, etc.), the approximatezoom level of the image and, importantly, information as to the type ortypes of damage depicted in the image 123, including the preciselocation of that damage (e.g., the pixels of the image 123 whichillustrate the damage of the particular damage type). Preferably, thetraining images 123 are annotated on a pixel by pixel basis so that eachpixel of the image 123 is annotated in the file 123A with each type ofdamage (which may be one type or multiple types) depicted by the pixel.Still further, the segmented training image files 124 may be files thatinclude images of unchanged (non-damaged) and changed (e.g. damaged)base objects illustrating the boundaries or outlines of various segmentsof a base object. In some cases, these segmented training image files124 can be images of different automobiles (of various different Y/M/Ms)that have been involved in different accidents and may also includeimages of non-damaged automobiles of different Y/M/Ms. Likewise, each ofthe segmented training image file 124 may include an information file124A associated therewith that provides information about the objectwithin the associated segmented training image 124, including, forexample, the Y/M/M of the vehicle depicted in the image 124, anindication as to whether or not damage is depicted or illustrated in theimage 124, the view or perspective of the object within the image 124(e.g., a side view, a right front corner view, etc.), the approximatezoom level of the image 124 and, importantly, an indication of the oneor more segments of the base object that are illustrated in the image124 (such as an outline or description or definition of the boundariesof each of the segments of the base object depicted in the image 124).Preferably, the training images 124 are annotated on a pixel by pixelbasis so that each pixel of the image 124 is annotated in the file 124Awith the segment of the object to which the pixel belongs. If desired,in some cases, the same images may be used for one or more of thetraining files 122, 123 and 124 as long as the proper damage areas,damage types and/or segments are identified for the images.

In a more particular example, each of the damage detection trainingimage files 122, damage-typed training image files 123 and segmentedtraining image files 124 may include one or more images of a damagedvehicle (conforming to one of the make/model/year types, for example, ofthe base object models 120 stored in the database 108). Generally, eachsuch damage detection training image file 122, damage-typed trainingimage file 123, and segmented training image file 124 may include one ormore digital photos taken of a particular automobile that has beendamaged in, for example, an accident. Such photos may be collected by,for example, owners of the automobiles depicted in the photos, anautomobile insurer against whom an insurance claim was made forrepairing or replacing the damaged automobile, etc. Still further, eachof the information files 122A, 123A and 124A may store other informationpertaining to the damaged automobiles in the training image files 122,123, 124 besides the damage location and type and segment location andtype information described above, such as the year/make/model andtrim-type of the damaged automobile, the country, state, city, zip code,and/or other geographical region in which the automobile was insured ordamaged, the mileage of the damaged automobile, the color of the damagedautomobile, the type of or location of the damage to the automobile,telematics data obtained from or about the damaged automobile associatedwith the accident, the parts which needed to be repaired or replaced asa result of the damage, the cost of repair or replacement of each suchpart, the type of damage to each such part, whether the automobile wasconsidered a total loss as a result of the damage, the cost of repair ofthe automobile if the automobile was repaired, the insurer of theautomobile, if any re-inspection was performed on the automobile duringrepair, capitation of the automobile, etc. Of course, other informationcould be stored for any or all of the training image files 122, 123,124, and the type of information stored for each of the training imagefiles 122, 123, 124 may vary depending on use, the type of object uponwhich change detection is to be performed, etc. Still further, while thetraining image files 122, 123 and 124 are described as separate files orimages, the information depicted therein may be combined into a singlefile, or may all be based on the same set of original images, if sodesired. As will be described in more detail herein, the base objectmodels 120, the segment models or files 121, the damage detectiontraining image files 122, the damage-typed training image files 123, thesegmented training image files 124 and the information files 122A, 123Aand 124A may be used by the image processing system 100, for example, toperform primary and secondary processing on photographs or images of anewly damaged automobile (referred to herein as a “target object” or a“target vehicle” or a “target automobile”) to determine the type ofand/or the extent of damage (change) and/or the precise location of thedamage (change) to the damaged automobile.

The server 112, which may include one or more microprocessors 128 andone or more computer readable memories 129, may store one or more imageprocessing or model training routines 130. The training routines 130 maybe implemented on the microprocessor 128 using the training images andsome or all of the data within the files 122, 122A, 123, 123A, 124 and124A to generate various other information or processing components usedin further image processing routines that analyze images of targetobjects (which correspond to one of the base object models 120) on whichchanges have occurred but for which changes have not been quantified andto quantify those changes in a more precise manner, such as bydetermining the type of change to the target object, the location of thechange on the target object, and/or the measurement and/or size of thechange on the target object.

In one example, one or more of the training routines 130 may implement amodel training routine using the training images 122, 123 and 124 andthe associated information files 122A, 123A and 124A to determine adifferent set of convolutional neural networks (CNNs) for use indetecting damaged areas of an object depicted in a set of target images,in detecting the type of damage depicted in a set of target images,and/or in detecting object segment boundaries depicted in a set oftarget images. In particular, one of the routines 130 may be used totrain a statistical model or a classification model that generallydetects damage to a target object to produce a “heat map” of the targetobject illustrating or indicating where damage exists within a targetimage based on the indication and description of damage depicted in theset of damage training images 122 and associated information files 122A.Here, the model training routine 130 may determine a first convolutionalneural network (CNN) 132 that is to be used by a model-based orclassification-based image processing routine to identify damagelocations within a new set of target images. The CNN 132 (which includesCNN coefficients to be used in a CNN-based classification model) areillustrated in FIG. 1 as being stored in the database 110, but thesecomponents could be stored in other memories associated with the imageprocessing system 100 if desired. Additionally, one of the routines 130may be used to train a statistical model that generally detects damagetypes in a target object to produce an indication of the type of damageillustrated or present in a target image, based on the indication anddescription of damage types depicted in the set of damage-typed trainingimages 123 and associated information files 123A. Here, the modeltraining routine 130 may determine a second convolutional neural network(CNN) 133 that may be used by a model-based or classification-basedimage processing routine to identify damage types (e.g., at varioussegments, areas, or locations of the target object) of detected damagewithin a new set of target images. The CNN 133 is illustrated in FIG. 1as being stored in the database 110, but this component (which includesa set of CNN based coefficients to be used in a CNN-based classificationmodel) could be stored in other memories associated with the imageprocessing system 100 if desired. Still further, one of the routines 130may be used to train a statistical model that generally detects segmentsand segment boundaries in a target image to produce an indication of thename (or type) of the segment and the boundaries of the segmentillustrated or present in a target image, based on the indication anddescription of segments and segment boundaries depicted in the set ofsegmented training images 124 and associated information files 124A.Here, the model training routine 130 may determine a third convolutionalneural network (CNN) 134 that is used by a model-based orclassification-based image processing routine to identify segments andsegment boundaries within a new set of target images. The CNN 134 isillustrated in FIG. 1 as being stored in the database 110, but thiscomponent could be stored in other memories associated with the imageprocessing system 100 if desired.

Thus, generally speaking, and as will be described in further detailherein, the image training routines 130 use the damage training images122 and information files 122A, the damaged-typed training images 123and information files 123A and the segmented training images 124 andinformation files 124A to produce and/or select the CNNs 132, 133 and134, in one example, that will be used by the image processing system100 to detect changes to target objects (such as to detect damage toautomobiles or other vehicles) and/or to detect the types of changes ordamage to automobiles or other objects, and/or to detect segments and/orsegment boundaries of damaged automobiles or other objects within imagesof these target objects. As described in more detail herein, a trainingtool 135, which may be stored in, for example, the server 112, mayoperate on a processor (such as the one or more processors 128) toassist a user in annotating the image files 122, 123 and 124 to providesome or all of the information within information files 122A 123A and124A.

Moreover, as illustrated in FIG. 1, the server 114, may include one ormore microprocessors 138 and one or more memories 139 that store ageneral image processing routine 140 that may perform image processingand image manipulation on a set of target images 142, which imagesdepict a target object represented by one of the base object models 120,but upon which change has occurred, to thereby detect changes to thetarget object, such as to detect or quantify damage to the target objectand one or more of a damage type, a damage location and a damage size orother damage descriptors. In this case, the target images 142, which maybe stored in the database 109 for example, represent the target objectas changed or altered with respect to the same object as represented byone of the base object models 120. Generally speaking, the imageprocessing routine 140 may use the CNNs 132, 133 and 134 (as stored inthe database or memory 110) as well as the set of target images 142 forthe target object to detect and quantify changes that have occurred tothe target object, where the changes are represented or depicted withinthe set of target images 142 of the target object. While the imageprocessing routine 140 is particularly described herein usingstatistical model that applies CNNs to images to determine occurrencesof damage or heat map, to determine damage types and/or to determinesegments and segment boundaries, any other type of statistical model orAI technique may be used to perform these functions instead or as well.

During operation, a user may log onto or access the system 100 via oneof the user interfaces 102 or 102A, may upload or store a new set oftarget images 142 of a target object in the database 109, and mayadditionally provide or store information in the database 109 related tothe new set of images 142, such as an identification of the targetobject within the new set of images 142 (e.g., the year/make/model andpotentially trim type of a damaged automobile depicted in the images),information about the target object (such as vehicle mileage, locationor geographical region of the automobile, etc.), as well as any otherdesired information about the images 142 or the target object within theimages 142, such as telematics data collected by the automobile depictedin the photographs, first notice of loss information as collected by theinsurance carrier of the automobile depicted in the photographs, anindication of the angle or perspective at which the object is depictedin the photograph, the approximate zoom of the image, etc. However, inone case, the user may simply upload the set of target images 142 andthe image processing system 100 may determine some or all of the neededimage information in the manners described herein. Still further, inanother embodiment, the system 100 may receive a set target images 142from a mobile device or another system (such as a First Notice of Loss(FNOL) system) and may process the images 142 on the fly or based on thereceipt of these images.

Of course, the new set of target images 142, potentially along withinformation related to the new set of target images 142, if provided,may be stored in the database 109 and/or provided to the database 109 inother manners, such as via a direct or indirect connection to a camera,via another device in the communication network(s) 104, e.g., via ahandheld device 102A connected to the network(s) 104 via a wirelessinterface 152, etc. Moreover, if desired, a user may use one of the userinterfaces 102, 102A to additionally or alternatively select a subset ofthe target images 142 that have been previously collected or taken ofthe target object, such as different views of the target object fromdifferent angles, and may provide these images to the database 109 ormay mark these images in the database 109 for use as and/or inclusion inthe new set of target images 142 to be processed by the image processingroutine 140.

The user may then initiate the image processing routine 140 to operateon the new set of target images 142 to detect changes within the targetobject depicted in the new set of target images 142 as compared to thebase object model 120 for that same object. Generally speaking, onceinitiated, the image processing routine 140 may use a first routine,referred to herein as an identifier routine or a Y/M/M routine 145 thatmay identify, from one or more of the set of target images 142, anidentification of the object depicted in the set of target images 142,such as the Y/M/M and/or trim type of an automobile or other vehicledepicted in the target images 142. The routine 145 may store theidentification information, such as the Y/M/M of the automobile withinor as part of the target images 142, e.g., in the memories 109

Thereafter, the image processing routine 140 may implement anotherroutine, referred to herein as a tagger routine 148, to determine imageinformation about each of the target images 142, such as the view orperspective of each target image 142 (e.g., a side view, a front view, acorner view), and the zoom level of each of the target images 142 (e.g.,no zoom, moderate zoom, high zoom), etc. The tagger routine 148 maystore view and zoom information obtained for or determined for each ofthe target images 142 as part of a tag for each of the target images 142and may store a set of tagged target images 149 that include the targetimages 142 along with the view, zoom and/or object identificationinformation (e.g., Y/M/M) of the object depicted in the correspondingtarget image, e.g., in the memories 139. In some implementations, thetagger routine 148 may store the tags separately from the target images142, and each tag may reference its corresponding target image 142. Invarious implementations, however, the identification information (Y/M/M)does not need to be part of the tag for the target images 142.

The routine 140 may also include a culling routine 150 that may processthe tagged target images 149, as tagged by the tagging routine 148, toselect a representative set of tagged target images 149 that best depictthe various views of the target object needed for later processing bythe routine 140. The culling routine 150 may store the culled or reducedset of target images as a selected set of tagged target images 152.Generally speaking, the culling routine 150 may process each of thetagged target images 149 to detect the quality of the images therein, todetect if any portion of the object (e.g., automobile) depicted thereinis occluded in the image, to determine if there is glare or otherphotographic detrimental effects within the image, to determine if theimage is at an appropriate zoom level (not too zoomed in or too faraway), to assure that the image depicts a view of the object orautomobile that is usable (e.g., that it is an exterior view of theautomobile), etc. The culling routine 150 may also or instead selectvarious ones of the tagged target images 149 to illustrate differentdesired or needed views of the object, so that each exterior side orportion of the object is depicted in at least one of the selected taggedtarget images 152. In any event, the culling routine 150 may produce areduced set or selected set of tagged target images 152 to be used inthe further processing steps in any other manner.

Additionally, the image processing routine 140 may include a heat mapperor heat mapping routine 160 that may process each of the culled orselected set of tagged target images 152 to produce, for each suchtarget image, a heat map mask (or heat map image) 162 illustrating theareas of the object in the associated target image that are damaged orthat are likely to be damaged. Thus, in one example, the heat mapperroutine 160 will produce for each of the selected set of tagged targetimages 152, a heat mask image 162 or map indicating which pixels of theimage 152 are indicative of damage and/or indicative of likely damageand/or a probability that damage has occurred at a respective locationof the object (e.g., automobile) represented by that pixel. Generallyspeaking, the heat mapper routine 160 will implement a statistical modelthat uses the CNN 132 to detect areas (e.g., pixels) of the targetobject depicted in the selected set of tagged target images 152 that aredamaged, and/or a probability of damage at that area (pixel). The heatmapping routine 160 may thus store a set of heat map masks or images 162that indicate, on a pixel by pixel basis, where damage exists in theimage or object and/or the probability of damage at a respective objectlocation denoted by each pixel.

Still further, the image processing system 140 may include a damage typedetector routine 170 that may process each of the selected set of taggedtarget images 152 and, in one example case, the areas of each of theselected set of tagged target images 152 that are indicated by the heatmap mask 162 for that image to include damage of some sort, using theCNN 133 in a CNN classification model to determine a type of damage ateach of the damage locations or at each of the pixels within the taggedtarget image being processed. More particularly, the damage typedetector 170 may implement a CNN-based classification model or routinethat produces, for each of the selected set of tagged target images 152,an image 172 that is labeled, on a pixel by pixel basis, with the typeor types of damage (if any) present at a respective location of theobject (e.g., automobile) represented by each pixel. Thus, all pixels ofa scratch depicted in one of the images 152 would be labeled with adamage type of “scratch,” while all pixels of a dent depicted in one ofthe images 152 would be labeled with a damage type of “dent,” and pixelsof the object at which no damage is located (as determined by the heatmap 162 for that image) would be labeled as “no damage” or somethingsimilar. Of course, in some cases, various pixels of the images 172 maybe labeled as being associated with or depicting multiple types ofdamage and, if desired, the damage type detector routine 170 may labelthe damage type with a detected probability of or likelihood orconfidence factor of that damage type (e.g., 60 percent likely that thispixel is associated with a scratch and 22 percent likely that this pixelis associated with a dent).

Moreover, the image processing routine 140 may include a segmentationroutine 180 that performs segmentation on each of the selected set oftagged target images 152 or, if desired and available, on each of thedamage-typed target images 172. In particular, the segmentation routine180 may determine one or more components or segments of the objectdepicted in each of the images 152 or 172 that this routine processes inorder to define the limits or boundaries of each of a set of knownobject segments as depicted in the images 152 or 172 being processed. Inthe situation in which the object is a vehicle or an automobile, thesegments may define predetermined or pre-established body panels of thevehicle (e.g., a hood, a passenger side front door, a rear bumper, afront grill, a wheel, a right front quarter panel, etc.). In this case,the routine 180 may process the image (including the information aboutthe view and zoom of the image) by implementing a CNN-basedclassification model or routine using the CNN 134 that produces, foreach of the selected set of tagged target images 152 or eachdamage-typed target image 172, a segmentation image or mask 182 that islabeled, on a pixel by pixel basis, with the vehicle segment to whichthe pixel belongs and, if desired, a probability or confidence factorthat the pixel belongs to that segment. The segmentation routine 180thus determines the segments that are depicted, either partially orfully, within the target images 152, 172 and the boundaries of thesesegments. In this manner, the segmentation routine 180 creates a set ofsegment images or segment masks 182 that (1) identify one or moresegments of the base object as depicted in the target image, and (2)identify the boundaries of the one or more identified segments asdepicted within the target image. These segment masks 182 are from thesame perspective as the camera angle of the target image and are sizedto the size of the target object depicted target image so that,advantageously, overlaying the segment mask 182 onto the selected taggedtarget image 152, 172 will define the segment boundaries for a number ofsegments of the object within the selected tagged target image 152, 172.

The image processing routine also includes a damage detailer routine 190that processes the selected set of tagged target images 152 or theselected set of tagged and damage-typed target images 172 with the heatmasks 162 and/or the segmentation masks 182 to determine particularchange or damage characteristics associated with the object and, inparticular, associated with each segment of the object. In this case,the damage detailer routine 190 may overlay one or more of the selectedtagged target images 152, or selected damage-typed tagged target images172 with the heat map mask 162 for that image and the segmentation mask182 for that image to determine how the damaged areas or pixels denotingdamage depicted within the image (identified by the heat map mask 162,for example) align with one or more of the segments (identified by thesegment map 182) within the image. The damage detailer routine 190 mayproduce a set of detailed files or images 192 that indicate more preciseinformation about the damage to each of the segment and, in particular,may identify each of the target object segments in an image that aredamaged, the precise size of the damage as compared to the base segment(e.g., the percent of the base segment that is damaged, the location ofthe damage with respect to the base segment, the size of the damage suchas the height, length and/or width of the damage as compared to the basesegment height, length and/or width and/or by using physicalmeasurements determined from the image processing, etc.), the type ofdamage if available from a damage-typed image 172, etc. The damagedetailer may include a number of routines such as a warping routine 195and a stitching routine 197 that enable the damage detailer routine 190to align various segments of the target object or automobile in thiscase with the damaged locations on the target object in a very precisemanner. In particular, the warping routine 195 may be used to warp athree-dimensional image into a two-dimensional image to make each partof the image have the same perspective or size and thus to reduce oreliminate three-dimensional effects within an image. This warpingroutine 195 is particularly useful when processing corner images of anautomobile (that illustrate part of the front or back and part of one ofthe sides of the automobile and thus have severe three-dimensionalperspective effects). This warping routine 195 may be applied in thesame manner to each of the associated corner images within the selectedset of tagged target images 152 or the selected set of damage-typedtagged target images 172, and to the heat map mask 162 and thesegmentation map or image 182 of each such selected tagged target imagein order to produce highly accurate damage information. Still further,the stitching routine 197 may be used to stitch various one of theimages 152 or 172 together so as to illustrate, in one image, a completesegment and/or an entirety of a damaged area. This stitching routine 197may be advantageously used when none of the selected tagged targetimages 152 illustrates a particular segment in its entirety or when itis desirable to provide higher zoom level detail of a particular segmentavailable within one image 152 within a second image 152 of the samesegment to provide higher damage resolution within parts of the stitchedimage, or when it is desirable to view or analyze a complete damage sitethat spans multiple segments.

While it will be understood that the image processing system 100 of FIG.1 is illustrated as a hardwired system having various servers anddatabases 106 connected to various user interfaces 102 via a hardwiredcommunication network 104, other processing and communicationconfigurations could be used as well or instead. For example, thenetwork 104 could be a wireless communication network, or could be acombined wired and wireless communication network, and the network 104may include any type of communication network, including a public localarea network (LAN) or wide area network (WAN), a private LAN or WAN, aset of direct communication connections, etc. Moreover, the network 104may include or use the Internet or the World Wide Web, to enable usersat the user interfaces 102 and 102A to communicate with the servers anddatabases 106 via an Internet connection to upload photos or images of atarget object for which change is to be detected to the database 109, toinitiate the image processing routines 130, 135 or 140, to view theresults of the routine 140, etc. Still further, while the servers anddatabases 106 are illustrated in FIG. 1 as including five differentdevices that have different information stored therein, any number ofservers, databases, and/or other computing devices could be used and, infact, the elements illustrated in the servers and databases 106 of FIG.1 could be stored within or distributed among more or less than fivedevices. For example, these components could be all stored in the samecomputer processing device, which could be the same computer processingdevice as one of the user interfaces 102 or 102A. In another example, atleast some (or all) of these components could be implemented using acomputing cloud architecture. Thus, the architecture of the imageprocessing system 100 of FIG. 1 could be changed to include more or lesscomputer devices connected in various different manners and stillfunction as described herein.

FIG. 2 is a block diagram of an exemplary image processing system 200(which may be the image processing routine 140 and/or the imageprocessing system 100 of FIG. 1) which processes images to detect,quantify, and characterize damages to an object, such as a vehicle.Generally speaking, the system 200 image processes an image of a damagedobject to automatically detect, quantify, and characterize damages tothe object depicted in the image without any or with very minimal humaninput, intervention, or guidance. The techniques of the system 200 maybe easily applied to images of any type of object, such as images of abuilding, a bridge, a movable structure, a road, a forest, jewelry,land, etc. However, for ease of discussion and not for limitationpurposes, the present disclosure discusses applying the techniquesdisclosed herein to images of a vehicle, such as an automobile, car,bus, truck, aircraft, heavy machinery, etc.

Generally speaking, the example system 200 processes an input or targetimage 202 of a damaged vehicle 205 by using a set of image processingmodules 210, 218, 225, 230, 238, 245, each of which image processes theinput image 202 to detect and/or determine different aspects of and/orinformation 215 associated with the damaged vehicle 205 depicted withinthe image 202. For example, when the system 100 includes an instance ofthe system 200, the image processing modules 210, 218, 225, 230, 238,245 may be included in or initiated by the image processing routine 140,and the information 215 may be stored in the memories 139. As isdescribed elsewhere within this disclosure, some of the modules of theset 210, 218, 225, 230, 238, 245 image process the input image 202 inconjunction with and/or by utilizing aspects and/or information 215determined by one or more of the other modules within the set, and somemodules of the set may image process the input image 202 withoututilizing any other additional information other than the input image202 itself. Further, although FIG. 2 visually depicts the modules 210,218, 225, 230, 238, 245 operating in a relatively sequential order, thisis only for ease of illustration and not limitation purposes. Forexample, in some implementations, two or more of the modules 210, 218,225, 230, 238, 245 may concurrently image process the input image 202 orinstances thereof, e.g., two or more of the image processing modules210, 218, 225, 230, 238, 245 may image process different instances ofthe input image 202 in parallel. In some implementations, one or more ofthe modules 210, 218, 225, 230, 238, 245 may not be applied to the inputimage 202 at all, and/or an additional, different image processingmodule (not shown) may be applied to the image 202 to discoverrespective information therefrom. Still further, for ease ofunderstanding, FIG. 2 is discussed with simultaneous reference to FIGS.3A-3G, each of which visually depicts one or more example input imagesdepicting a damaged vehicle and/or depicts various informationcorresponding to the images and/or to the damaged vehicle which has beendetected, determined, and/or output by one or more of the imageprocessing modules 210, 218, 225, 230, 238, 245.

In FIG. 2, the system 200 obtains the electronic image 202 of thedamaged vehicle 205 (which may be any of the target images 142 of FIG.1, for example, or which may be received via a network interface fromanother computing device or computing system). The electronic image 202typically is a digital image, and may be a two-dimensional or athree-dimensional image; however, for ease of illustration and notlimitation purposes, the present disclosure refers to the image 202 as atwo-dimensional, digital image. In some scenarios, the system 200obtains an indication of the year, make, and model 208 (which isinterchangeably referred to herein as the “Y/M/M” 208) of the vehicle205. The system 200 may obtain the Y/M/M 208 of the depicted damagedvehicle 205 via any suitable technique, such as via user input, byreading a file, by image processing the input image 202, etc. Forexample, the system 200 may include an optional Identifier module 210which image processes the target image 202 to determine the Y/M/M 208 ofthe vehicle 205 depicted within the target image 202. The Identifiermodule 210 may include a Y/M/M model 212 which has been speciallytrained (e.g., by training routine 130) on images of both damaged andundamaged vehicles to detect or determine the particular year, make,model, and optionally trim type of a vehicle depicted in the subjectimage. For instance, the Y/M/M model 212 may include one or moreanalytical or AI models which have been trained, by using any one ormore suitable machine learning technique(s), on images of damaged andundamaged vehicles that have been labeled with the respective Y/M/M andoptionally trim type of the vehicle depicted within each image. In oneexample implementation, the Y/M/M of the vehicle depicted within theimage may be determined using one or more of the image processingtechniques described in any of co-owned U.S. patent application Ser Nos.16/874,154, 16/874,167, or 16/874,175, the disclosure of each of whichis hereby expressly incorporated by reference herein. In any event, thevehicle 205 and the vehicle's Y/M/M 208 may be stored 215 or otherwisemade available for use by the image processing modules 218, 225, 230,238, 245 of the system 200. This information may be stored, for example,as part of the information file 124A of FIG. 1.

To illustrate using an example, FIG. 3A depicts an exampletwo-dimensional image 302 of a damaged vehicle 305 which may be inputinto and image processed by the system 200, as well as depicts anindication of the vehicle's Y/M/M 308. In FIG. 3A, the Y/M/M 308 of thevehicle 305 has been previously determined by another AI model or system(not shown) which has been specially trained to detect or determine theY/M/M of vehicles depicted in images. For example, one or more modelsand/or techniques described in co-owned U.S. patent application Ser.Nos. 16/874,154, 16/874,167, or 16/874,175 may have been applied to theimage 302 to determine its Y/M/M 308. In embodiments, the imageprocessing system 200 includes the AI model or system 210 whichdetermines a depicted vehicle's Y/M/M, if desired.

In some embodiments, the system 200 includes a Tagger module 218 whichimage processes the input image 202. The Tagger module 218 includes atagging model 220 which has been specially trained (e.g., by trainingroutine 130) on images of both damaged and undamaged objects, such asvehicles, to detect or determine the particular view of an object orvehicle depicted in a subject image, and optionally to detect ordetermine a degree or amount of zoom with respect to the depicted objector vehicle. For example, the tagging model 220 may include one or moreanalytical or AI models which have been trained, by using any one ormore suitable machine learning technique(s), on images of damaged andundamaged vehicles that have been labeled with the respective view ofthe vehicle depicted within each image (e.g., side or planar views suchas Left Side View, Right Side View, Front View, and Back View;perspective or corner views such as Left Front Corner View, Right FrontCorner View, Left Rear Corner View, and Right Rear Corner View; interiorviews such as Dashboard View, and Back Seat View; and/or other suitabletype(s) of views of the depicted vehicle). In some implementations, thetagging model 220 includes one or more of the models described in U.S.Pat. No. 10,319,035 (the disclosure of which is hereby expresslyincorporated by reference herein), and the Tagger module 218 utilizesone or more techniques described in U.S. Pat. No. 10,319,035 todetermine or detect the particular view of the damaged vehicle 205depicted within the image 202. In other implementations, the Taggermodule 218 may additionally or alternatively utilize other suitabletechniques, if desired. Additionally, in some scenarios, the taggingmodel 220 may operate on both the input image 202 and the Y/M/M 208 ofthe depicted vehicle 205 as inputs. Further, in some embodiments, thetagging model 220 (or another model included in the Tagger module 218,not shown) may include one or more analytical or AI models which havebeen trained, by using any one or more suitable machine learningtechnique(s), on training images of damaged and undamaged vehicles whichhave been labeled with respective degrees, levels, or amounts of zoom(e.g., Negligible Zoom, Average Zoom, High Zoom, and/or other suitablelevel(s) of zoom). The Tagger module 218 applies the tagging model 220(and applies the separate zooming model, if so implemented in the Taggermodule 218) to the input image 202 to thereby determine or detect theparticular view 222 a of the vehicle 205 depicted in the image 202, andoptionally to determine or detect the degree, level, or amount of zoom222 b of the image 202 with respect to the vehicle 205 depicted in theimage 202. The Tagger module 218 tags or labels the input image 202 withan indication of the detected view 222 a and an indication of thedetected level of zoom 222 b, and the tags 222 a, 222 b corresponding tothe image 202 are stored or otherwise made available 215 for use byother modules 210, 225, 230, 238, 245 of the system 200.

To illustrate, FIG. 3B depicts the example image 302 of the damagedvehicle 305 of FIG. 3A, where the image 302 has been image processed bythe Tagger module 218 and consequently tagged as being a Right FrontCorner perspective view 310 and as having Negligible Zoom 312. As alsoshown in FIG. 3B, each of two other example images 320, 322 of thedamaged vehicle 305 have also been tagged by the Tagger module 218 asbeing, respectively, a Right Front Corner perspective view 325, 328 ofthe damaged vehicle 305. The Tagger module 218 has tagged the image 320as being an “Average Zoom” 330 image, whereas the Tagger module 218 hastagged the image 322 as being a “High Zoom” 332 image.

Returning back to FIG. 2, in some embodiments, the system 200 includes aHeat Mapper module 225 which image processes the input image 202. TheHeat Mapper module 225, which may be the heat mapper routine 160 of FIG.1, includes a heat mapping model 226 which has been specially trained(e.g., by training routine 130) on images of both damaged and undamagedobjects, such as vehicles, to detect or determine the presence of damage(and/or the probability or the confidence level of the presence ofdamage) to various parts, portions, segments, sites, or locations of anobject or vehicle depicted in a subject image. In an example, the heatmapping model 226 includes one or more analytical models or AI modelswhich have been trained by using any one or more suitable machinelearning technique(s) on training images (e.g., training images 122)which have been labeled on a per pixel basis with a respectiveindication of the presence or absence of actual damage at acorresponding part, portion, segment, and/or site of a vehiclerepresented by each pixel. The Heat Mapper module 225 applies the heatmapping model 226 to the input image 202, thereby generating a heat mapmask 228 (which may be the heat map masks 162 of FIG. 1, and which isalso interchangeably referred to herein as a “heat map” or “heat mapimage” 228) of the depicted vehicle 205. Each colored pixel of the heatmap mask 228 represents a respective location on the vehicle 205 atwhich the heat mapping model 226 has discovered the presence of damageto the vehicle 205 (and/or has discovered that the probability of thepresence of actual damage to the vehicle 205 is greater than apredetermined threshold), and each different color corresponds to adifferent degree or amount of damage and/or a different probability ofthe presence of actual damage at the respective location on the vehicle205. As indicated above, the Heat Mapper module 225 may implement astatistically based characterization routine and, in particular, acharacterization routine that implements a CNN model to analyze eachpixel of an input image 202 to determine a probability that damageexists at that pixel. The operation of this model will be described inmore detail with respect to FIG. 4A. The Heat Mapper module 225 may thencreate a heat map 228 of the image that details or annotates, for eachpixel of the input image, if damage has been detected and, if desired,the probability of the detected damage. The Heat Mapper module 225 may,if desired, apply a thresholder to the damage probability determined ateach pixel so that any determined potential damage less than thethreshold is detected as no damage. However, in other implementations,the heat mapping model 226 includes one or more of the models andperforms the techniques described in U.S. Pat. No. 9,886,771 (thedisclosure of which is hereby expressly incorporated by referenceherein), and the Heat Mapper module 225 implements one or more of thetechniques described in U.S. Pat. No. 9,886,771 to generate a heat mapcorresponding to a damaged vehicle depicted in an image. In otherimplementations, the Heat Mapper module 225 may additionally oralternatively utilize other suitable techniques. Additionally, in somescenarios, the heat mapping model 226 may operate on both the inputimage 202 and the Y/M/M 208 of the depicted vehicle 205 as inputs. Thegenerated heat map or heat map mask 228 of the vehicle 202 may be storedor otherwise made available 215 for use by other modules 210, 218, 230,238, 245 of the system 200.

To illustrate, FIG. 3C depicts as an example usage of the Heat Mappermodule 225. In FIG. 3C, the system 200 has applied the Heat Mappermodule 225 to each of images 302, 320, 322, thereby generatingrespective heat map masks 340, 342, 345 which have been respectivelyoverlaid on their respective target images 302, 320, 322. The colorscale of the heat map masks 340, 342, 345 uses the darkest red color toindicate the highest degree or amount of damage and/or the highestprobability of the presence of damage, and gradually changes the colorsto brighter red, orange, yellow, green, and in some cases, blue torespectively indicate decreasing levels of amounts of damage and/ordecreasing probabilities of the actual presence of damage at respectivelocations on the vehicle 305 represented by corresponding pixels of theheat map masks 340, 342, 345. The embodiments of the heat map masks 340,342, 345 shown in FIG. 3C do not mask any pixels of respective images302, 320, 322 which corresponds to locations on the object or vehicle atwhich no damage was detected by the heat mapping model and/or at which aprobability of the presence of actual damage to the vehicle is below apre-determined threshold. That is, the pixels of the heat map masks 340,342, 345 which do not correspond to any damage to the vehicle arerendered without color or as clear/transparent.

Returning again to FIG. 2, in some embodiments, the system 200 includesa Damage Typer module 230 (which may be the damage typer routine 170 ofFIG. 1) to image process the input image 202. The Damage Typer module230 includes a damage typing model 232 which has been specially trained(e.g., by training routine 130) on images of damaged (and potentiallyundamaged) objects, such as vehicles, to detect or determine one or moredifferent types of damage of various parts, portions, segments, and/orsites of an object or vehicle depicted in a subject image (and/or todetermine the probabilities of one or more different types of damage atthe various parts, portions, segments, and/or sites of the depictedobject or vehicle). For example, the damage typing model 230 may includeone or more analytical models or AI models which have been trained byusing any one or more suitable machine learning technique(s) on trainingimages (e.g., training images 123) which have been labeled on a perpixel basis with respective one or more different types of damage whichare physically present at the sites or locations on vehicles representedby each pixel. In one case, the Damage Typer module 230 may implement astatistically based characterization routine and, in particular, acharacterization routine that implements a CNN model to analyze eachpixel of an input image 202 to determine a probability that damageexists at that pixel. The operation of this model will be described inmore detail with respect to FIG. 4B. The Damage Typer module 230 maythen create a damage-typed image or mask that details or annotates, foreach pixel of the input image 202, the type or types of damage detectedat that pixel 235 a and, if desired, the probability of the detecteddamage type(s) 235 b. The Damage Typer module 230 may, if desired, applya thresholder to the damage type probability determined at each pixel sothat any determined potential damage type less than the threshold isdiscarded or detected as no damage of that damage type. Thus, the DamageTyper module 230 applies the damage typing model 232 to the input image202, thereby generating, for each of a set of sites on the vehicle 205depicted within the image 202, a respective set of types of damage 235 aand/or respective probabilities 235 b of the actual presence of thedifferent types of damage at each site. Examples of different types ofdamage which may be detected by the Damage Typer module 230 include butare not limited to:

Dents—a type of damage where the surface of a vehicle is significantlydeformed. Dents may happen to metal, plastic, and/or rubber materials ofa vehicle.

Buckles—a type of damage where a segment or panel of a vehicle iscompletely deformed or bent under pressure. Typically, when viewing abuckle from outside of the vehicle, a buckle creates a large concavearea with hard, clearly delineated edges, and a peak of a buckle may bevisibly distinguished when viewing the buckle from the interior of thesegment or panel. As such, a “buckle” may be more severe than a “dent.”Some buckles may cause certain segments or panels of a vehicle (e.g.,hood, trunk lid, etc.) to deform outwards instead of or in addition todeforming in the concave manner. Buckles are typically associated withmetal materials of a vehicle, and most cases of buckle damage are notrepairable (e.g., a buckled portion of the vehicle typically isreplaced).

Creases—a type of damage which is a linear (or is an essentially linear)dent. Typically creases are one inch or greater in length, and may occurmost commonly on metal panels or segments of vehicle.

Dings—a type of damage which is a small dent of a small curvature,typically dime-or nickel-sized and not visible from a distance.

Scratches—a type of damage including a cut or a mark which has aone-dimensional or two-dimensional shape, and typically may have beencaused by rubbing the surface of the vehicle with a sharp object.Scratches may be categorized into at least three different typesdepending on the depth of the scratch, for example:

a clear coat scratch may be a low-depth or light scratch on an outerlayer (e.g., the clear coat layer) of the vehicle;

a color coat scratch may be a medium-depth scratch which penetrated theclear coat layer of the vehicle and removed paint color from thevehicle; and

a primer scratch may be a deep-depth scratch which exposes bare metal ofthe vehicle.

Cracks—a type of damage in which a segment or a panel of a vehicle hasbeen broken apart due to impact.

Tears—a type of damage in which a segment or panel of a vehicle hassplit, typically occurring in plastic, rubber, or fabric materials.

Holes—a type of damage which is a cavity in a panel or segment,typically caused by a sharp penetration of a small object, and occurringless commonly than Tears.

Bend—a type of damage which includes a gradual change in shape of thevehicle between the damaged area and an undamaged area.

Misalignment—a type of damage characterized by a visible gap within asegment or panel of a vehicle.

Missing—a type of damage which is characterized by a missing part,panel, or segment of a vehicle.

Of course, other type of damages may be additionally or alternatelydetected by the Damage Typer module 230. Further, demarcations betweenclassifications of different types of damages (e.g., dents vs. buckles,cracks vs. tears, and the like) and/or classifying multiple types ofdamages which are simultaneously present at a same site (e.g., dent andscratch, crack and misalignment and scratch, etc.) may be defined duringlabeling of the training images. Accordingly, the Damage Typer module230 may detect multiple types of damages (and/or probabilities ofmultiple types of damage) occurring at a particular portion, segment, orsite of the depicted damaged vehicle 205. Additionally, in someembodiments, the Damage Typer module 230 may respectively apply thedamage typing model 232 to different portions, segments, or sites of thevehicle 205 depicted within the image 202 to detect the respectivedamage types (or probabilities of the respective presences thereof) ateach different portion, segment, or site of the vehicle 205 depictedwithin the image 202. The Damage Typer module 230 may indicate thedetected damage types 235 a by listing the respective type(s) ofdetected damages of various different portions, segments, or sites ofthe vehicle, by indicating the respective probabilities 235 b of actualdamage types occurring at the various different portions, segments, orsites of the vehicle, and/or by using any other suitable indication. Thedetermined damage type(s) 235 a of the vehicle 202 and/or probabilitiesthereof 235 b may be stored or otherwise made available 215 for use byother modules 210, 218, 225, 238, 245 of the system 200.

FIG. 3D illustrates an example usage of the Damage Typer module 230. InFIG. 3D the system 200 has applied the Damage Typer module 230 to eachof the images 302, 320, and 322, and the Damage Typer module 230 hasdetected that each of the images 302, 320, 322 respectively depicts twodamage types, e.g., “dent” and “scratch,” as denoted respectively by thereferences 360, 362, 365.

Returning still to FIG. 2, in some embodiments, the system 200 includesa Segmenter module 238 (which may be the segmentation routine 180 ofFIG. 1) to image process the input image 202. The Segmenter module 238includes a segmenting model 240 which has been specially trained (e.g.,by training routine 130) on images of damaged (and potentiallyundamaged) objects, such as vehicles, to detect or determine differentsegments or predefined portions of an object or vehicle depicted in asubject image. Typically, the different segments may be delineated ordefined based on a base object model (such as the segmentation model 121of FIG. 1) or based on downstream uses of the information 215 providedby the image processing system 200, e.g., may be delineated or definedbased on consumers of the information 215 which is generated and/ordiscovered by the image processing system 200. For example, theinformation 215 provided by the image processing system may be utilizedto order parts needed to repair the damaged vehicle, and accordinglysegments of the vehicle may be defined according to sets or groups ofparts which may be ordered and/or replaced as a whole, such as a frontbumper system, a rear right side panel, a driver's door and windowsystem, a wheel assembly, etc. In some implementations, segments of thevehicle may be defined as different views of the vehicle which can bedepicted within a single image, e.g., driver's or left side view,passenger's or right side view, front view, back view, etc. As such, inthese implementations, a perspective or corner view of a vehicle may besegmented into two different views, e.g., a back view and a right sideview, a front view and a left side view, etc.

Accordingly, the segmenting model 240 may include one or more analyticalmodels or AI models which have been trained by using any one or moresuitable machine learning technique(s) on training images (e.g.,training images 124) which have been labeled on a per pixel basis with arespective indication of the respective segment of a vehicle which isdepicted by each pixel. In one embodiment, the Segmenter module 238 mayimplement a statistically based characterization routine and, inparticular, a characterization routine that implements a CNN model toanalyze each pixel of an input image 202 to determine a predefinedsegment to which that pixel belongs (e.g., which the pixel depicts) and,in some cases, a probability or confidence factor that the pixel of theimage belongs to (e.g., depicts) that segment. The operation of thismodel will be described in more detail with respect to FIG. 4. TheSegmenter module 238 may then create a segmented image that details orannotates, for each pixel of the input image 202, which segment orsegments of the object the pixel belongs (e.g., depicts) and, ifdesired, a probability or confidence factor associated with thatclassification. The Segmenter module 238 may, if desired, apply athresholder to the segment classification probability determined at eachpixel so that any determined segment classification probability lessthan the threshold is discarded.

Thus, the Segmenter module 238 applies the segmenting model 240 to theinput image 202, thereby generating a segmentation mask 242 (which isinterchangeably referred to herein as a “segmentation map,”“segmentation map mask,” “segmentation map image,” or a “segmentationimage” 242) of the segments of the vehicle 205 which are depicted withinthe image 202. Different segments of an object may be depicted withinthe segmentation map as mutually exclusive areas, and the differentsegments depicted within the images may be indicated by respectiveborders, colors, and/or other suitable indicators. Each differentsegment may be depicted within the segmentation mask 242 (e.g., on apixel basis) by using a different color, in an embodiment. In someembodiments, in addition to generating the visual segmentation mask 242of the image 202, the Segmenter module 238 may generate a list ofvehicle segments which are at least partially depicted within the targetimage 202 of the vehicle 205. The generated segmentation mask 242 of thevehicle 205 may be stored or otherwise made available 215 for use byother modules 210, 218, 225, 230, 245 of the system 200.

FIG. 3E illustrates an example usage of the Segmenter module 238. InFIG. 3E, the system 200 has applied the Segmenter module 238 to theimage 302 of the vehicle 305 to obtain a corresponding segmentation mask370, and the system 200 has applied the Segmenter module 238 to theimage 320 of the vehicle 305 to obtain a corresponding segmentation mask372. Advantageously, the Segmenter module 238 may denote a same segmentof the vehicle 205 which is depicted in the different images 302, 320 ofa same vehicle 305 by using the same color in both segmentation masks370, 372. For example, the segmentation map mask 370 and thesegmentation map mask 372 both depict a front right wheel segment usinga first color, as denoted by references 375 a, 375 b, and bothsegmentation masks 370, 372 depict a right front headlight segment usinga second color, as denoted by references 378 a, 378 b.

In FIG. 2, the system 200 includes a Damage Detailer module 245 (whichmay be the damage detailer routine 190 of FIG. 1) which determines andquantifies or measures the respective sizes and/or locations of damagedareas or sites on the vehicle 205 depicted within the image 202. TheDamage Detailer 245 includes a Warping module or Warper 248 (which maybe the warping routine 195 of FIG. 1) which operates on images whichhave been tagged as being a corner or perspective view, or thatotherwise are determined to be an image of the vehicle which depictsmultiple perspective lines along the vehicle in multiple differentdirections with respect to the location of the camera which captured theimage, and/or other types of three dimensional aspects of the vehicle.(When a subject image is not a corner or perspective view, does notdepict multiple directions of different lines of sight, or otherwisedoes not depict aspects of the vehicle in essentially three dimensions,such as when the subject image depicts only a single planar view of thevehicle, the Damage Detailer module 245 does not apply the Warper 248 tothe subject image, and may proceed directly to determine measurements,e.g., as is described in later sections below.) Generally speaking, theWarping module 248 performs a warping technique on the image 202 toflatten the image (e.g., to make a two-dimensional view of the object asdepicted in the image to account for or compensate for three dimensionalaspects of the object depicted in the image), thereby generating awarped image 202′. The Warping module 248 may apply the warpingtechnique to the image based on a set of warping waypoints depictedwithin the image. In some embodiments, the set of warping waypoints maybe predefined, for example, the outside corners and/or boundaries of apanel, window, or other segment of the depicted object; physicallocations at which different panels or segments of the depicted objectmeet (e.g., the intersection of a window edge, a door top edge, and adoor hinge edge); a physical location of a part (e.g., the attachmentlocation of a rearview mirror); etc. In some embodiments, the set ofwarping waypoints utilized to warp the subject image 202 may bedynamically determined or discovered so that the warping of the image202 is optimized. For example, the Warping module 248 may utilize amachine-learning optimization technique to discover a set of optimumwarping waypoints for the image 202. At any rate, upon warping orflattening the image 202, the steps or actions taken by the Warpingmodule 248 to perform the warping or flattening may be saved ormaintained as a warping transformation “WT” associated with the image202 which results in the warped image 202′. That is, WT(image 202)=image202′.

The warping transformation WT which the Warping module 248 used togenerate the warped image 202′ may be applied to the heat map mask 228to generate a warped heat map mask 228′ of the image 202 (e.g., WT(heatmap mask 228)=heat map mask 228′), and may be applied to thesegmentation map mask 242 to generate a warped segmentation map mask242′ of the image 202 (e.g., WT(segmentation map mask 242)=segmentationmap mask 242′). In some embodiments, the Damage Detailer 245 initiallyapplies the Warping module 248 to the segmentation mask 242 of the image202 to generate the warped segmentation mask 242′ and determine thewarping transformation WT, and subsequently uses the determined warpingtransformation WT to warp the heat map mask 228 and optionally to warpthe image 202, thereby generating the warped heat map mask 228′ andoptionally the warped image 202′.

To illustrate with an example, in FIG. 3F, the Damage Detailer module245 has applied the Warping module 248 to the segmentation map 370 ofthe vehicle 305 depicted in the image 302 (where the segmentation map370 includes the Right Side panel segment 380), thereby generating aflattened segmentation map 370′ in which the right side panel segment380 is depicted in a two dimensional representation 380′, and therebydetermining the warping transformation “WT₃₇₀” corresponding totransforming the original segmentation map 370 into the warpedsegmentation map 370′. As also shown in FIG. 3F, the Damage Detailermodule 245 has applied the warping transformation WT₃₇₀ to the image 302of the vehicle 305 (which depicts the right side panel 382 of thevehicle 305), and has generated a flattened image 302′ in which theright side panel 382 depicted in the image 302 is depicted in a twodimensional representation 382′. Further, in a similar manner, althoughnot shown in FIG. 3F, the Warping module 248 may additionally oralternatively warp the heat map 340 of the vehicle 305 depicted in theimage 302 by using the same warping technique and transformation WT₃₇₀used by the Warping module 248 to generate the warped heat map 340′.

The Damage Detailer 245 may include a Stitcher module or Stitcher 250(which may be the stitching routine 197 of FIG. 1) which operates on twoinput images, each of which depicts a respective portion or entirety ofa same particular component or segment of the vehicle 205, to generatean aggregate, stitched image 252 which includes information and featuresof the particular component or segment from each of the two inputimages, e.g., a single, composite or panoramic image. The two inputimages may depict different portions of the particular segment which maypartially overlap, and/or one of the input images may have a greaterdegree of zoom compared to the other one of the input images. TheStitcher 250 may convert or transform one or both of the two images sothat both images have a common or normalized uniform depth of field,which may require warping one or both the images (e.g., when one or bothof the images depict perspective views). The Stitcher 250 may generatethe stitched image 252 by aligning one or more stitching waypoints orfeatures of the object which are depicted in both images, for example.Additionally or alternately, the Stitcher 250 may generate the stitchedimage 252 by aligning one or more stitching waypoints or features of theparticular component or segment depicted in both images, of respectivesegmentation maps, and/or of other segments or portions of the vehiclewhich are depicted in both of the two images, which may be particularlyuseful when one of the images depicts the particular component orsegment with a greater level of zoom. In an embodiment, the Stitcher 250may identify the depicted stitching waypoints within the two images(whether the waypoints are segment waypoints, segmentation mapwaypoints, or other commonly depicted waypoints) by utilizingconvolution, k-dimensional tree filtering, and/or other suitabletechniques. In an embodiment, at least some of the stitching waypointsmay be predefined, for example, the outside corners of a panel, window,or other segment of the depicted object; a physical location at whichdifferent panels or segments of the depicted object meet (e.g., theintersection of a window edge, a door top edge, and a door hinge edge);a physical location of a part (e.g., the attachment location of arearview mirror); etc. It is noted that the set of stitching waypointsand the set of warping waypoints discussed herein may be mutuallyexclusive sets or may at least partially intersect.

At any rate, based on the steps and actions applied by the Stitcher 250to the two input images to generate the stitched image 252, the Stitcher250 may determine the corresponding stitching transformation “ST”, andthe Stitcher 250 may apply the stitching transformation ST to otherimages related to the two target input images, such as heat map images,segmentation images, and/or the target images, thereby respectivelygenerating a stitched heat map, a stitched segmentation map, and/or astitched target image.

At any rate, and returning to FIG. 2, the Damage Detailer module 245measures the respective sizes 255 and/or locations 258 of damaged areasof the vehicle 205 depicted within the image 202 or within the stitchedimage 252 by utilizing the heat map mask 228 in conjunction with thesegmentation map mask 242 of the image 202, 252 (or, when the image 202is a corner view, by utilizing the warped heat map mask 228′ inconjunction with the warped segmentation map mask 242′). In particular,the Damage Detailer 245 determines a depth of field indicator of thesegmentation map 242 based on the Y/M/M 208 of the vehicle 205. Thedepth of field indicator typically is in units of pixels per unitlength, such as pixels per centimeter, pixels per meter, etc., where thelength is scaled to the actual physical size of the physical vehicle305. The Damage Detailer 245 overlays the heat map mask 228 and thesegmentation map 242 of the image 202 (or, if appropriate, overlays thewarped heat map mask 228′ and the warped segmentation mask 242′ of theimage 202, or overlays respective stitched images) to thereby determinewhich pixels of segments of the vehicle 205 which are included in thesegmentation map 242, 242′ represent damage to the vehicle 205 atcorresponding locations on the physical vehicle 205, e.g., as indicatedby the colored pixels of the overlaid heat map mask 228. Additionally,the Damage Detailer 245 measures the respective size 255 of eachdetected damaged area and/or measures a respective distance 258 of thedetected damaged area (e.g., with respect to an edge of a segment orother waypoint of the vehicle 205) by using the depth of fieldindicator. A more detailed description of the Damage Detailer 245 isprovided elsewhere in this disclosure.

Accordingly, in view of the above, when the image processing system 200applies all of the modules 210, 218, 225, 230, 238, 245 to the inputimage 202, the image processing system 200 automatically detects thepresence of damage at a site or area on the vehicle 205 which isdepicted in the input image 202, and automatically determines a measuredsize 255 and/or measured location 258 of damaged area, as well as thetypes 235 a of damage which were incurred at the area, and/or types ofinformation 215 corresponding to the image 202 and/or to the damagedvehicle 205. Of course, in embodiments, the system 200 need not applyall modules 210, 218, 225, 230, 238, 245 to the image 202. For example,the system 200 may only apply the Damage Typer module 230 to the image202 to determine the types of damage (or probabilities thereof) depictedwithin the image 202. In another example, the system 200 may omitapplying the Tagger module 218 to the image 202, and instead maydetermine whether or not the image 202 requires warping by utilizinganother technique.

In yet another example, the system 200 may apply the Tagger module 218(e.g., to determine the respective depicted view of the object withinthe input image 202 and within other input images, such as front,driver's or left side, passenger's or right side, back, particularperspective, etc.), and may omit the Segmentation module 238. In thisexample, the Damage Detailer 245, including the Warper 248 and theStitcher 250, may operate on the different input images in theirentireties instead of operating on different depicted segments of theobject. In still another example, the system 200 may apply theSegmentation module 238 to some of the input images, forgo applying theSegmentation module 238 to others of the input images, and apply theDamage Detailer 245 (including the Warper 248 and the Stitcher 250) asappropriate. Other embodiments in which the system 200 applies fewerimage processing modules to the target image 202 and its associatedinput images or applies additional or alternate image processing modulesto the image 202 and associated images are possible.

Further, some input images 202 may not be processed by any of themodules 225-245 of the system 200. For example, prior to or upon thetagging of a particular target image 202 by the Tagger module 218, aculling routine associated with the image processing system 200, such asculling routine 150 of FIG. 1, may determine that the particular targetimage 202 is to be culled from the set of target images of the damagedvehicle. For instance, the culling routine may cull the target image 202due to the target image 202 being a duplicate view of the damagedvehicle, due to image quality, due to a too near or too far away levelof zoom of the target image 202, etc. In these situations, the targetimage 202 is not further processed by the system 200.

In some embodiments, the image processing system 200 (or instancesthereof) may operate on each of a set of target images depicting a samedamaged vehicle, and information discovered by the system 200 indicativeof or relating to damage depicted within each image (e.g., damagelocations, damage types, damage sizes, and other damage characteristics,such as but not limited those described above) may be aggregated andscreened to remove any duplicate information (which may have beenrespectively obtained from image processing each of multiple images of asame area of damage, for example), and provided to a user interface(e.g., user interface 102), stored in memory (e.g., the memories 139,the database 109, or some other memories), and/or transmitted via one ormore networks to a recipient computing device. Different instances ofthe system 200 may operate on at least some of the set of target imagesin parallel, if desired, and/or a single instance of the system 200 mayoperate on at least some of the set of target images sequentially.

FIGS. 4A-4C depict block diagrams of various different CNN model baseddetectors that each implement a different CNN to (1) detect damage to avehicle within an image (FIG. 4A), (2) detect a damage type of a damagesite on a vehicle within an image (FIG. 4B), or (3) detect one or moresegments of a vehicle within an image (FIG. 4C). In particular, FIG. 4Adepicts a CNN model based characterization engine 402 that may be usedby the Heat Mapper routine 225 of FIG. 2 to develop a heat map of anobject within an image 404 (which may be any of the selected taggedtarget images 142 of FIG. 1 or target images 202 of FIG. 2), to identifythe areas of likely damage in that image 404. In particular, as will beunderstood, the characterization engine 402 includes or uses a CNN 406(which may be the CNN 132 of FIG. 1) to analyze or process a set ofmodel inputs 408 to produce, for each set of inputs 408 present at themodel 402, a set of outputs 410 which are indicated, in this case, as anoutput 410A that indicates whether damage exists based on the currentset of inputs 408 and as an output 410B that indicates a probabilitythat damage exists (or a confidence factor) that damage exists based onthe current set of inputs 408.

Generally speaking, the model inputs 408 include a first input thatspecifies or identifies the view of the image 404 (e.g., front view,side view, corner view, right rear corner view, left rear corner view,passenger side view, etc.) Typically this view will be one of a set ofknown or predetermined or enumerated views that the image can fallwithin and the values of this input may be limited to the standard eightviews of an automobile (e.g., four corner view, two side views, a frontview and a rear or back view). Still further, the model inputs 408include a second input that specifies or identifies an approximate zoomlevel of the image (e.g., no zoom, moderate zoom, high zoom, etc.).Again, the zoom level input is typically limited to one of a fixed setof possible approximate zoom levels. The view and zoom level inputs 408may be obtained from an information file associated with the image 404being processed, may be entered by a user or may be determinedautomatically using other techniques described herein. In any event,these inputs have been determined to be very useful in obtaining highlyaccurate damage indications or damage outputs for the model 402 whichimplements a CNN that has been trained on images with this informationpresent. Moreover, model inputs 408 include a set of pixel values 412from the image 404 being processed. This set of pixel values 412 willtypically be a fixed set of contiguous pixels in the image 404 and arethe pixels defined by an input template 414 defining a square, arectangular, a circle, an oval, or any other desired shape that isscanned through the image 404. The size and shape of the input template414 may also be provided as an input to the model 402. The inputtemplate 414 is focused or centered about a center pixel 412A (whichdoes not actually need to be in the center of the template 414) forwhich the model 402 is determining the damage outputs 410A and 410B.

During operation of the model 404, the Heat Mapper routine 225 moves theinput template 414 to be centered on a particular center pixel 412A ofthe image 404, applies the pixel values 412 at the various locationscovered by the input template 414 to the model inputs 408, andimplements model calculations using the CNN 406 on those input pixelvalues and the view and zoom inputs 408, to produce the outputs 410A and410B. The implementation of a CNN is well known and thus will not bedescribed in detail herein. However, as is known, the CNN 406, ifproperly trained, represents or contains a set of factors or weights tobe used in the model calculations that have been determined, in atraining process, to provide the best statistical estimation of thedamage estimation output 410A based on the processing (using a similarinput template 414) of a large number of training images in which thedamage estimation or value at each pixel is known or quantified.Generally speaking, the training algorithm or process implements arecursive mathematical calculation (also referred to sometimes as aregression algorithm) that determines the set of CNN factors or weightswhich, when implemented in the model 402, provides the best (moststatistically accurate) estimation of the known damage outputs in theset of training images over the entire set of training images.

In any event, after calculating the outputs 410A and 410B, the model 402stores the output values 410A and 410B in a heat map image 416 ascorresponding to or as associated with the current input center pixel412A within the input template 414, and then moves the input template414 over a new center pixel 412A. The model 402 then calculates theoutput values 410A and 410B for that new center pixel 412A and storesthese outputs in the heat map image 416 for that new center pixel 412A.The model 402 repeats this process to scan the entire input image 404 todetermine for each pixel therein, a set of output values 410A and 410Bdefining the existence of damage at that pixel and the probability orlikelihood for that damage calculation. As CNNs are generally known, itwill be evident to those of ordinary skill in the art that the model 402can produce, as part of the calculations therein, a confidence factor orprobability factor indicating how closely the predicted output 410A isstatistically correlated with the training data. Moreover, the size ofthe CNN or input template 414 may be chosen to be any desired size suchas a 4 by 4, 10 by 10, 100 by 100, 50 by 75, etc. Generally speaking,the higher the number of pixel inputs leads to a more computationallyexpensive model (and a harder model to train) but may result in higheraccuracy. Thus, in any event, the model 402 comprises a characterizationengine and, more particularly, a CNN based image model that processeseach of the pixels of a selected image to determine the particularpixels of the image 404 (or of the object within the image 404) that aredamaged, and the likelihood of damage (e.g., the confidence of thatcalculation). In this case, the characterization engine or CNN model 402uses a CNN transform 406 that has been developed or trained using atraining engine that analyzes a plurality of images of objects (e.g.,different automobiles) damaged in various different manners which havebeen annotated, on a pixel by pixel basis, to indicate which pixels ofthe training image represent damaged areas of the objects (and whichimages have also been annotated to indicate the view and/or zoom levelof the image). Importantly, it is not necessary that the CNN trainingroutine or the model 402 know the Y/M/M of the vehicles depicted in thetraining images or the image 404 being processed.

Still further, FIG. 4B depicts a CNN model based characterization engine422 that may be used by the Damage Typer routine 230 of FIG. 2 todevelop a damage-typed image of an object within the image 404 (whichmay be any of the selected tagged target images 142 of FIG. 1 or targetimages 202 of FIG. 2), to identify the areas of likely damage in thatimage 404 and, additionally to detect a type of damage in each of thedamaged areas or sites. In particular, as will be understood, thecharacterization engine 402 includes or uses a CNN 406 (which may be theCNN 133 of FIG. 1) to analyze or process a set of model inputs 408 toproduce, for each set of inputs 408 present at the model 402, a set ofoutputs 430A and 430B. In this case, the output 430A indicates a type ofdamage at a particular pixel based on the current set of inputs 408 andthe output 430B indicates a probability or confidence factor that thedetected damage type is correct. Here, because the model 402 ischaracterizing the pixel as falling within a no damage category, or oneof a plurality of different but limited number of damage types, theoutputs 430A may include multiple different damage types, and theconfidence factor outputs 430B may include a different confidence factorfor each of the potentially detected damage types. Essentially, in thiscase, the model 402 determines, based on the inputs 408 to the model402, the likelihood that the pixel 412A being analyzed falls within eachof the possible damage types (or no damage) and, as such, calculates aprobability or correlation factor that the pixel falls into each of thedifferent possible damage types, including no damage. In any event, themodel 402 may select, as the damage type outputs 430A, all of the damagetypes and may provide the confidence factor for each damage type on theoutputs 430B, may select only the damage types that have a confidencefactor over a particular threshold and may provide the confidence factoron the output 430B for each of those damage types that are over theparticular threshold or may select only one damage type with the highestconfidence factor and provide that damage type and confidence factor onthe outputs 430A and 430B. In other cases, the model 402 may simplyoutput the damage type that has the highest confidence factor orcorrelation value. In any event, it will be understood that a singlepixel of the image 404 may be determined to fall within zero, one, twoor even more different damage types.

Of course, similar to the model of FIG. 4A, the model inputs 408 includea first input that specifies or identifies the view of the image 404(e.g., front view, side view, corner view, right rear corner view, leftrear corner view, passenger side view, etc.) and a second input thatspecifies or identifies an approximate zoom level of the image (e.g., nozoom, moderate zoom, high zoom, etc.) Moreover, the inputs 408 of FIG.4B include a set of pixel values 412 from the image 404 being processed.This set of pixel values 412 will typically be a fixed set of contiguouspixels in the image 404 as defined by an input template 414, and theinput template may define a square, a rectangular, a circle, an oval,etc. and be of any desired size. The size and shape of the inputtemplate 414 may also be provided as an input to the model 402. Theinput template 414 is centered about or focused on a pixel 412A forwhich the model 402 is determining the damage-typed outputs 430A and430B. This area may be any particular or desired size and shape and thesize or shape may be determined by the training algorithm which trainsthe model CNN 426.

In a manner similar to that described with respect to FIG. 4A, theDamage Typer routine 230 moves the input template 414 centered over aparticular center pixel 412A, applies the pixel values 412 at thevarious locations around the center pixel 412A as defined by the inputtemplate 414 to the model 402, and implements the model calculationsusing the CNN 426 on those input pixel values and the view and zoominputs 408, to produce the outputs 430A and 430B. Again, theimplementation of a CNN is well known and thus will not be described indetail herein. However, as is known, the CNN 426, if properly trained,represents or contains a set of factors or weights to be used in themodel calculations that have been determined, in a training process, toprovide the best statistical estimation of the damage type estimationoutput 430A based on processing of a large number of training images(using the same input template 414) in which the damage type at eachpixel is known or quantified. Generally speaking, the training algorithmor process implements a recursive mathematical calculation (alsoreferred to sometimes as a regression algorithm) that determines the setof CNN factors or weights which, when implemented in the model 402,provides the best (most statistically accurate) estimation of the knowndamage types at each pixel in the set of training images over the entireset of training images.

In any event, after calculating the outputs 430A and 430B, the model 402stores the output values 430A and 430B in a damage-typed image 436 ascorresponding to or as associated with the current input center pixel412A within the input template 414 and moves the input template 414 overa new center pixel 412A. The model 402 then calculates the output values430A and 430B for that new center pixel 412A and stores these outputs inthe damage-typed image 436 for that new center pixel 412A. The model 402continues to scan the entire input image 404 using the input template414 to determine for each pixel therein, a set of output values 430A and430B defining the type of damage at that pixel (if any) and theprobability or likelihood or confidence factor for that damage typecalculation. As CNNs are generally known, it will be evident to those ofordinary skill in the art that the model 402 can produce, as part of thecalculations therein, a confidence factor or probability factorindicating how closely the predicted output 430A is statisticallycorrelated with the training data. Moreover, the size of the CNN orinput template 414 may be chosen to be any desired size. In any event,the model 402 comprises a characterization engine and, moreparticularly, a CNN based image model that processes each of the pixelsof a selected image to determine a damage type (if any) associated witheach of the particular pixels of the image (or of the object within theimage), and a confidence factor or probability of the determination. Inthis case, the characterization engine or CNN model 402 uses a CNNtransform 426 that has been developed or trained using a training enginethat analyzes a plurality of images of objects (e.g., differentautomobiles) damaged in various different manners which have beenannotated, on a pixel by pixel basis, to indicate which pixels of thetraining image represent damaged areas of the objects and the type ofdamage at those areas (and which images have also been annotated toindicate the view and/or zoom level of the image).

Still further, FIG. 4C depicts a CNN model based characterization engine402 that may be used by the Segmenter routine 238 of FIG. 2 to develop asegmented image of an object within the image 404 (which may be any ofthe selected tagged target images 142 of FIG. 1 or target images 202 ofFIG. 2), to identify the segments depicted within that image 404. Inparticular, as will be understood, the characterization engine 402includes or uses a CNN 446 (which may be the CNN 134 of FIG. 1) toanalyze or process a set of model inputs 408 to produce, for each set ofinputs 408 present at the model 402, a set of outputs 450A and 450B. Inthis case, output 450A indicates a segment name or identity to which aparticular pixel belongs based on the current set of inputs 408, and theoutput 450B indicates a probability or confidence factor associated withthe detected segment identity. In this case, because the model 402 ischaracterizing each pixel as falling within one of various differentsegment types (or no segment at all), the output 450A may includemultiple different segment identities for a particular pixel and theconfidence factor output 450B may include a different confidence factorfor each of the potentially detected segments. Essentially, in thiscase, the model 402 determines, based on the inputs 408 to the model402, the likelihood that the center pixel 412A of the input template 414falls within each of the possible segment types and, as such,essentially calculates a probability or correlation factor that thepixel 412A falls into each of the different possible segment types,including no segment (e.g., a pixel could be background imageinformation). As such, the model 402 may select, as the segment typeoutputs 450A, only the segment types that have a confidence factor overa particular threshold and may provide the confidence factor on theoutput 450B for each of those segment types that are over the particularthreshold. In other cases, the model 402 may simply output the segmenttype that has the highest confidence factor or correlation value on theoutputs 450A and 450B. In any event, it will be understood that a singlepixel of the image 404 may be determined to fall within zero, one oreven two different segment types based on the confidence factorscalculated by the model 402.

Of course, similar to the model of FIG. 4A, the model inputs 408 includea first input that specifies or identifies the view of the image 404(e.g., front view, side view, corner view, right rear corner view, leftrear corner view, passenger side view, etc.) and a second input hatspecifies or identifies an approximate zoom level of the image (e.g., nozoom, moderate zoom, high zoom, etc.) Moreover, the inputs 408 of FIG.4C include a set of pixel values 412 from the image 404 being processedas defined by an input template 414. The inputs and input template 414is similar in nature to that described for FIG. 4A and FIG. 4B, but neednot be the same size or the same shape.

In a manner similar to that described with respect to FIG. 4A, theSegmenter routine 238 moves the input template 414 centered on aparticular center pixel 412A, applies the pixel values at the variouslocations around the center pixel 412A as defined by the input template414, and implements the model calculations using the CNN 446 on thoseinput pixel values and the view and zoom inputs 408, to produce theoutputs 450A and 450B. Again, the implementation of a CNN is well knownand thus will not be described in detail herein. However, as is known,the CNN 446, if properly trained, represents or contains a set offactors or weights to be used in the model calculations that have beendetermined, in a training process, to provide the best statisticalestimation of the segment type estimation output 450A based onprocessing of a large number of training images (using the same size andshape input template 414) in which the segment type at each pixel isknown or quantified. Generally speaking, the training algorithm orprocess implements a recursive mathematical calculation (also referredto sometimes as a regression algorithm) that determines the set of CNNfactors or weights which, when implemented in the model 402, providesthe best (most statistically accurate) estimation of the known segmentsin the set of training images over the entire set of training images.

In any event, after calculating the outputs 450A and 450B, the model 402stores the output values 450A and 450B in the segmented image 456 ascorresponding to or as associated with the current input center pixel412A within the input template 414 and moves the input template 414 overa new center pixel 412A. The model 402 then calculates the output values450A and 450B for that new center pixel 412A and stores these outputs inthe segmented image 456 for that new center pixel 412A. The model 402operates to scan the entire input image 404 (or most of the input image404) to determine for each pixel therein, a set of output values 450Aand 450B defining the identity of the segment of the object present atthat pixel and the probability or likelihood or confidence factor forthat segment identification calculation. As CNNs are generally known, itwill be evident to those of ordinary skill in the art that the model 402can produce, as part of the calculations therein, a confidence factor orprobability factor indicating how closely the predicted output 450A isstatistically correlated with the training data. Moreover, the size ofthe CNN or input template 414 may be chosen to be any desired size andshape.

FIG. 4D depicts a block diagram of a training module 460 that may beused to produce or create the CNNs 406, 426 and 446 used in the models402 of FIGS. 4A-4C from the same or from various different sets oftraining images. In particular, the training module 460 may include aprocessing engine 462 that ultimately processes a set of training images470 using a set of information files 472 associated with each of thetraining images 470, wherein the information files include informationas to the zoom and view of the image 470 and as to a known property oroutcome of each pixel in the image (a damage indication, a damage typeor a segment identity with which each pixel is associated). The trainingor processing engine 460 uses known CNN modeling techniques to determinea CNN 480 that, when applied to the same training images 470 in one ofthe models 402 of FIGS. 4A-4C provides the best or most statisticallyaccurate estimate of the known property or outcome (e.g., whether anyparticular pixel of an image 470 is damaged, what type of damage anyparticular pixel is associated with, or what segment a particular pixelis associated with) over all of the pixels of the entire set of trainingimages 470.

As will be understood, the training module 460 may take, as inputsthereto, a CNN size (e.g., the number of pixel inputs to the CNN) and aCNN shape (the template shape of the pixels in an input template orscanning template), e.g., rectangular, square, circular, etc. Stillfurther, the training module 460 may take as inputs at any particulartime, the zoom level and view of the image being processed (i.e., thezoom level and view of the current one of the images 470 beingprocessed) as well as the pixel values for the pixels defined by aninput template of the specified size and shape centered over or focusedon a current pixel or center pixel for which a prediction is known. Thetraining module 460 may also receive, as inputs, the known property oroutcome values of each of the input pixels within the training image 470being processed, e.g., damage or not damaged, a type of damage, or asegment to which the pixel belongs. This known outcome or propertyinformation may be stored in and received from a set of informationfiles 470A for each of the training images 470.

As will be understood, the engine 462 implements a recursive algorithmthat recursively develops a set of weights or factors for the CNN 480.In particular, the engine 462 implements any of various knownmathematical procedures or techniques to develop a set of CNN weightsthat, when used in a CNN model such as one of the CNN models 402 ofFIGS. 4A-4C, predicts the known outcome or property for each image pixelin each of the set of training images 470 in the most statisticallyaccurate manner (overall). Generally speaking, the engine 462 uses aparticular set of CNN weights to determine a statistical measure of howaccurate these weights are in predicting the known properties oroutcomes of each input pixel for each pixel of each training image 470.The routine then alters the weights using known recursive techniques andrepeats the prediction calculations over all of the pixels of all of thetraining images to determine if the new set of weights is more or lessaccurate in making the predictions, in a statistical sense. The engine462 then continues to alter the CNN weights in a recursive manner, andtests those weights, to ultimately develop a set of CNN weights thatbest performs or that is most accurate in predicting the known outcomes,in a statistical sense, over the entire set of pixels in the entire setof training images. When the model trainer 460 finds a CNN that performswell or is otherwise acceptable, the engine 462 outputs this CNN to beused by one of the models 402 of FIGS. 4A-4C. Of course, if desired, thetraining module 460 may repeat the CNN calculations using differentinput template sizes and/or shapes to determine an input template sizeand shape that best suited for making the desired predictions ordetections. Of course, the input template size and shape effects thenumber of CNN weights that are within the CNN being developed.

FIG. 5 illustrates a model training tool 500 that may be implemented bythe training module or routine 135 of FIG. 1 to assist a user increating the training images 470 (which may be any of the trainingimages 122, 123 and 124 of FIG. 1) and the information files 472associated therewith (which may be any of the information files 122A,123A and 124A of FIG. 1). In particular, the tool 500 includes acommunications module or interface (which may include hardware and/orsoftware) 510, a user interface module 512 and an annotation module orroutine 514 which operate together to enable a user to annotate, on apixel by pixel basis, each of a set of non-annotated training images 520to create an information file 520A for each of the training images 520,wherein the particular information within the information file 520A maybe used by the training module 460 of FIG. 4D (along with the images520) to create one of the CNNs 480. Here, the image files may be any ofthe training images 122, 123 and 124 of FIG. 1 or the training images470 of FIG. 4D, and the information files 520A may be any of theinformation files 122A, 123A and 124A or the information files 472 ofFIG. 4D. Generally, the communications interface 510 may be any desiredtype of communications interface (implemented in hardware and/orsoftware and/or firmware) that communicates via any desired or availablecomputer network or computer connection (such as a wired or wirelesscommunication network, a memory device such as a flash or hard drivememory, etc.) to access the set of training images 520 wherever theseimages are stored. Generally speaking, the training images 520 may be aset of images of various different objects (automobiles) with differenttypes and amounts of change (damage) thereon (and in some cases, objectswith no change or damage). The objects, in this case, will be describedas vehicle or automobiles, but could be other types of objects instead.Thus, in one case, the non-annotated images 520 may be images ofdifferent automobiles (of various Y/M/Ms) with different parts thereofdamaged in different manners. It is preferable that the training images520 include a plurality of examples of each type of automobile (Y/M/M)that will be or that could be analyzed by the models 402 of FIGS. 4A-4Cand that the training images 520 include damages of each type that willbe predicted by the model 402 of FIG. 4B. Moreover, it is preferablethat the training images 520 includes different views of theseautomobiles (e.g., side, front and corner views) as well as images ofdifferent zoom levels (e.g., no zoom, moderate zoom and high zoom).

The user display interface 512 is generally an interface that connectsto and drives a number of user input/output devices, including a userdisplay device 530 and one or more user input devices such as akeyboard, a mouse, an electronic pen, a touch screen, etc. Stillfurther, the annotation module 514 may be a computer routine executed ona processor (not shown in FIG. 5A) that controls and uses thecommunications interface 510 to access new training images 520, toenable a user to view and annotate each of the training images 520 usingthe user input/output devices 530, 532 and to store the annotated imagesor files 520A in a further memory device, not specifically shown in FIG.5A but which could be in any of the databases or servers of FIG. 1,including the database 108.

Generally, the training tool 500 of FIG. 5A implements the annotationroutine 514 to assist a user in annotating each of the training images520 to create the information files 520A. More generally, the annotationroutine 514 may use the communication module or interface 510 to accessa particular non-annotated training image 520 and may display thatnon-annotated training image 520 on the user display device 530. Theannotation routine 514 may then enable a user to indicate, on thedisplayed training image 520, using an electronic pen, a touch screen, akeyboard, a mouse or track pad, or any other type of selector or inputdevice 532, one or more of (1) a set of pixels within the displayedimage 520 that is associated with or that depict damage to the vehicle,(2) a set of pixels within the displayed image 520 that is associatedwith a particular type of damage and/or (3) a set of pixels within thedisplayed image 520 that is associated with a particular segment of thevehicle depicted in the image (e.g., associated with a particularautomobile panel). The annotation routine or module 514 may enable auser, for example, to draw a line around the pixels in the displayedimage associated with damage in general, pixels associated with aparticular damage site of a particular type of damage, such as a dent ora scratch depicted in the displayed training image, or associated with aparticular vehicle body panel or segment within the displayed image, toindicate the pixels in the training image 520 that depict damage ingeneral, that depict damage sites of a particular type and/or thatdepict particular segments of a vehicle. To make pixel selection easier,the annotation routine 514 may enable a user to fill in areas of theimage (e.g., inside or bounded by the line drawn by the user) to therebyselect or associate each pixel in the filled in area. The annotationroutine 514 may then enable the user to associate or label the selectedset of pixels of the displayed training image 520 as being damaged (orillustrating damage), as being associated with a particular type ofdamage (such as with a dent, a scratch, a hole, a tear, etc.) and/or asbeing associated with a particular segment of the object (such as with aparticular body panel of a vehicle), via the user interface input/outputdevices 530 and 532. As an example, annotation routine 514 may enablethe user to label or associate a particular type of damage or aparticular segment of an object depicted within the image 520 with theselected pixels using a drop down menu presented on the display screen530, and the annotation engine or routine 514 may then mark the trainingimage 520 and, in particular, the selected individual pixels of thetraining image 520, with a particular damage type or a particularsegment as identified by the user or model trainer to create or updatean information file 520A associated with the displayed training image520. In some cases, the user may also be able to indicate the view ofthe image (e.g., right front corner view, side view, back view, etc.)and/or an approximate zoom level of the image (no zoom, moderate zoom,high zoom, etc.) via the user input/output devices 530, 532, which zoomand view information can also be stored as part of the information file520A for that image. These preprocessed images 520, which correspond tothe damage training images, damage-typed training images and segmentedtraining images discussed previously, along with the information files520A therefor, can then be advantageously used to train statistical orclassification models (in one example, CNN models) that are then used toprocess pixels in new images (e.g., target images) (1) to detect damageor a probability of damage at each of those pixels, (2) to detect a typeof damage and/or a probability of a particular type of damage at each ofthose pixels and/or (3) to detect the segment of the object and/or aprobability of the segment of the object to which each of those pixelsbelong.

In one example, the annotation routine 514 may implement a routine 570of FIG. 5B to assist a user in performing training image annotation andcreation. In this case, at a block 572, the routine 570 may obtain oneof the non-annotated training images 520 from a memory (using thecommunication module or interface 510 of FIG. 5A). The routine 570includes a block 574 which may then display this image 520 on a userinterface device such as a display screen 530 of FIG. 5A. As an example,FIG. 6A illustrates a user interface display screen 600 that may beprovided by the routine 570 and that may be used to provide imageinformation to a user and to enable a user to annotate the images asdescribed herein. In the example of FIG. 6A, the display screen 600includes an image display area 602 (in which a current non-annotatedtraining image 520 is displayed) and an annotation area 604 (in which auser can enter or provide information to annotate the image displayed inthe image display area 602).

Referring again to FIG. 5B, a block 576 of the routine 570 next enablesthe user to select and mark an area of the displayed image in the imagedisplay area 602 (e.g., to specify a set of pixels in the displayedimage) as being associated with a site of interest within the displayedimage. This site of interest can be, in one example, any damage site(e.g., specifying the pixels of the image illustrating one or moredamaged areas of the object within the image), can be, in anotherexample, a damage site of a particular type (e.g., specifying the pixelsof the image illustrating a damage site of the object within the imageof a particular damage type, for example, a dent or a scratch) and canbe, in still another example, a particular segment or body panel of theobject within the image (e.g., specifying the pixels of the imageillustrating or associated with a particular segment or body panel ofthe automobile within the image). As an example, the block 576 maydisplay in the annotation area 604 in the display screen of FIG. 6A oneor more sets of controls that assist the user in specifying a set ofpixels of the image within the image display area 602 to be associatedwith or grouped as a site of interest. For example, the block 576 maydisplay a box 610 that includes controls that enable the user to specifythe manner in which the image in the image display area 602 may bemarked with a pen or other marking element (e.g., a finger on a touchscreen, a mouse selection, etc.) For example, the box 610 may include aset of controls 612 that enable the user to specify a pen or strokesize, a color or opacity of the marker, etc. The box 610 may alsoinclude a set of controls 614 that enable the user to switch to aneraser mode (to erase markings in the image area 602) to clear the imagemarkings, to undo a recent action, to unselect one or more selected ormarked pixels, etc. The controls 614 may also include a control 616 thatenables a user to fill in pixels within a marked bounded area on theimage within the display area 602.

In one case, the block 576 may enable a user to select different sets ofpixels within the image in the image display 602 associated withmultiple different sites of interest. For example, the block 576 maydisplay or use, in the annotation area 604 of FIG. 6A, a box 620 thatenables a user to add or specify or create, in this case, differentsites of interest in the image area 602, as it is possible that theremay be multiple sites of interest in a single image. Thus, in theexample screen 600 of FIG. 6A, the user may manipulate a Site 0 region630 to indicate that the user is annotating a first damage site, and theroutine 570 may then present a set of controls for the Site 0 site ofinterest to enable the user to mark and annotate the first site ofinterest (e.g., a first damage site). In the case of FIG. 6A, theannotation routine 570 is being used to mark damage types and so, inthis case, the routine 570 enables the user to specify, using thecontrols 610, the area or areas (pixels) within the image displayed inthe image area 602 that are to be associated with the first damage site.The user may, for example, use an electronic pen to circle a set ofpixels in the image area 602 associated with the first damage site Site0. FIG. 6B illustrates the screen 600 of FIG. 6A in which the user hasused the controls 612 and an input device to specify a site of interestpresent in the image currently being displayed in the image display area602 by painting the pixels (marked with reference number 625) in thesite of interest. The block 576 of FIG. 5 could as well or insteadenable a user to circle or outline the selected set of pixels and to usethe box 616 in the annotation area 604 to fill in the circled oroutlined pixels to mark each of the outlined or circled pixels (i.e.,the pixels within the marked area) as being associated with the firstdamage site Site 0.

In any event, after a user has specified a set of pixels within theimage display area 602 associated with a particular site of interest(which can be, for example, damage of any type, damage of a particulartype, or a segment of the object in the image), a block 578 of theroutine 570 of FIG. 5B enables a user to annotate the selected pixels ofthe site of interest with specific information about those pixels, suchas that the pixels in the site of interest represent damage to theobject in the image area 602, represent damage to the object of aspecific or particular type, or represent a specific segment or bodypanel of the object displayed in the image area 602. For example, in thedisplay screen 600 of FIG. 6B, the user may use an input block 632 tospecify the type of damage associated with the marked first damage siteSite 0. The block 578 may, for example, provide a drop down menu 627 (asillustrated in FIG. 6C) or other control (e.g., a separate window) onthe screen 600 to enable the user to specify which of a predeterminedset of damage types (or segment types if the image is being annotated toillustrate segments) is to be associated with each of the marked orselected pixels of damage site Site 0. In the case of FIG. 6C, the userhas specified that the site of interest is associated with a damage typeof “crease.”

In any event, when the correct pixels of a site of interest are marked,and the correct damage type or segment information for each of themarked pixels is specified, a block 580 of the routine 570 of FIG. 5Bmay determine if there are more sites of interest in the image beingdisplayed in the image area 602. For example, the block 580 may detectif a user uses the control 620 to add or specify a second damage site toindicate yet another damage site on the image displayed within the imagearea 602. If so, the routine 570 may return to the block 576 and enablesthe user to repeat the steps of using the control panel 614 to mark thepixels of the second site of interest Site 1 and the controls 630associated therewith to specify the damage or other information for theselected pixels (e.g., a damage type or a segment type). FIG. 6Dillustrates a display screen in which multiple damage sites have beenmarked and annotated with different damage types. In one embodiment,hovering over the damage site icons for the different sites of interestmay cause the routine 514 to display the pixels associated with thatsite of interest.

Still further, the routine or module 514 and in particular one of theblocks 576 or 578 of FIG. 5B may assist the user in selecting damagesites and segments by estimating the locations of different damage sitesand/or the boundaries of different segments or body panels within theimage displayed in the image display area 602 and enabling a user toview and alter those estimations via the pixel selection tools describedherein with respect to FIGS. 6A-6D and/or to further annotate thoseestimations. In particular, the routine 514 may implement one or more ofthe techniques disclosed in U.S. Pat. No. 9,886,771 that analyze animage using a known base model to identify damage areas and/or segments.As will be seen, these techniques determine and use landmarks in animage to determine a more precise definitions of or estimations of thecamera angle and zoom level of the image, and then rotate and size abase image of the object (e.g., a base image of an undamaged automobileof the same Y/M/M determined from a two-dimensional or three-dimensionalbase model object) to the same perspective and view as the image beingprocessed. These techniques can then perform heat mapping andsegmentation outlining in the manner described in U.S. Pat. No.9,886,771. Upon performing these processes on an image that is beingdisplayed in the image display area 602, the routine 570 can thenoverlay these heat maps and/or segment maps onto the image in the imagedisplay area 604 to provide the estimated damage areas or segments. Forexample, in FIGS. 6A-6D, a set of controls 633 indicate a set ofpredicted damage sites in the image displayed in the image area 602(e.g., five predicted damage sites DMG0 to DMG4). The routine 570 mayenable a user to hover over or select these controls 602 to view thepredicted areas of damage associated with each predicted damage site.The user may use the controls in the annotation area 604 to change theestimated set of pixels, fill-in bounded areas of pixels, specifyparticular ones of the predicted damage pixels and/or different pixels,etc. as being associated with one or more of the sites of interest(e.g., Site 0, Site 1, etc.). These predicted sites of interest may beused to assist the user in selecting and filling out the pixelsassociated with the different actual sites of interest. In some cases,the user may accept the pixels of a predicted site of interest as a newsite of interest.

When the block 580 determines that a user is finished marking pixels ofdifferent sites of interest (e.g., damage sites or segment sites) andannotating each set of marked pixels with a damage type for a damagesite or a segment for a segment site, the user may select the Submit 634on the display screen 600 and a block 582 of the routine 570 may savethat information for the image in an information file associated withthe image. Additionally, at any time, the routine 570 may enable a userto delete one of the sites of interest using a delete button 536provided for each site of interest.

Moreover, if desired, the routine 570 may include a block 584 thatenables a user to enter or specify the view of the image within theimage area 602 and/or the zoom level of the image within the image area602. These values may again be provided from a predetermined orenumerated list of possible values provided in a drop down menu or otherinput box provided to the user. Of course, in some cases, the images mayalready be annotated with the view and zoom level information and so theblock 582 may not be needed. Moreover, after the image information for aparticular training image is stored, a block 586 of the routine 570 maydetermine if there are more images to be annotated. For example, in theexample of FIG. 6A, the controls 640 may be used to enable a user toskip an image (e.g., select the next image to view) or to go back to aprevious image. Of course, the routine may repeat the blocks 572-586 forany number of training images.

It will be noted that, while the example screen displays of FIG. 6A-6Dillustrate the annotation of images for damage sites of different damagetypes, the module 514 and the routine 570 may be used in a similarmanner to specify and select pixels associated with damage sites ingeneral (without specifying a damage type) and may be used to specifyand identify segments of an automobile or other object, such as bodypanels of an automobile.

FIG. 7 depicts a block diagram of an example damage detailer method 700which may be utilized to process images to determine a precise sizeand/or location of damage to an object. For example, the method 700 maybe implemented by the damage detailer routine 190 of FIG. 1 or by theDamage Detailer module 145 of FIG. 2. For ease of discussion, though,and not for limitation purposes, the method 700 is discussed below withsimultaneous reference to FIGS. 1, 2, and 3A-3G.

At a block 702, the method 700 includes obtaining a segmentation mapimage and a heat map image of a target digital image of a damagedvehicle. The segmentation and heat map images may have been generatedfrom the target digital image, e.g., in manners such as previouslydescribed. For example, the obtained segmentation map 242 and theobtained heat map image 228 may have been generated from a targetdigital image 202, or the obtained segmentation map 370 and the obtainedheat map image 340 may have been generated from a target digital image302. In some scenarios, the segmentation map and/or the heat map may beobtained from a data store, such as data store 109 or memories 139. Insome scenarios, the method 700 includes generating the segmentation mapand/or the heat map from the target digital image. For example, themethod 700 may utilize one or more techniques described elsewhere withinthis disclosure to generate the segmentation map 370 and/or the heat map340 from the target digital image 302.

At a block 705, the method 700 includes determining whether or not theobtained images depict a perspective view of the damaged vehicle. Forexample, the method 700 may access a tag indicative of the type of viewof the target digital image (e.g., a tag generated by the tagger routine148 or by the Tagger module 218), the method 700 may initiate anapplication of the tagger routine 148 or the Tagger module 218 to thetarget digital image to determine the type of view depicted therein, orthe method 700 may utilize some other technique to determine whether ornot the obtained images include a perspective view of the damagedvehicle.

When the segmentation map image and the heat map image depict aperspective view of the vehicle (e.g., the “Yes” leg of block 705), at ablock 708 the method 700 includes warping the segmentation map and theheat map to thereby generate a warped segmentation map and a warped heatmap. Generally speaking, warping the segmentation map and heat mapimages may include transforming each of the segmentation map image andthe heat map image depicting one or more varying depths of field alongone or more perspective lines into a corresponding warped or flattenedmap image which has a uniform (or essentially uniform) depth of fieldacross at least a planar portion of depicted vehicle, if not most oreven all of the depicted vehicle. A depiction of a planar portion of adepicted object having an “essentially” uniform depth of field, asutilized herein, generally refers a depiction in which a majority of thepixels or other portions of the depiction have a common depth of field,where the majority may be defined by a threshold, for example. As thesegmentation map image and the heat map image have been generated from asame target digital image (and thus are from the same perspective as thecamera angle of the target image and are of the same size), the samewarping transformation is applied to each map image to generate itsrespective warped map image. The method 700 may utilize any suitabletechnique for warping images such as, for example, the warpingtechniques described elsewhere within this disclosure, e.g., thetechniques discussed in more detail with respect to FIGS. 2, 8A, and 8B.Indeed, in an embodiment, the block 708 may incorporate at least aportion of the method 820 of FIG. 8B.

Upon or at some time after warping 708 the segmentation and heat maps,or when the segmentation and heat maps do not depict a perspective viewof the vehicle (e.g., the “No” leg of block 705), at a block 710 themethod 700 includes determining the (uniform or essentially uniform)depth of field indicator corresponding to the target digital image fromwhich the (potentially warped) segmentation and heat maps weregenerated. In an embodiment, determining 710 the depth of fieldindicator of the target image includes determining a respective depth offield indicator corresponding to each vehicle segment of a plurality ofvehicle segments depicted in the target image, which may be an entiretyor a subset of the vehicle segments depicted in the target image. In anembodiment, the respective depth of field indicators may be determinedbased on the segmentation map and is described herein as such. However,in other embodiments, the respective depth of field indicators mayeasily be determined based on the target image itself. At any rate, therespective depth of field indicator corresponding to each of theplurality of vehicle segments may be determined based on a comparison(e.g., a ratio or a relative relationship) of a distance betweenselected or pre-defined measurement waypoints as depicted within thesegmentation map and an actual, measured distance between correspondingwaypoints of a physical vehicle corresponding to the depicted damagedvehicle. In an example, a particular depth of field indicator may bedetermined based on a comparison of a length of the principal axis ofeach vehicle segment as depicted within the segmentation map and anactual measurement of the length of a principal axis of a correspondingvehicle segment of a physical vehicle corresponding to the depicteddamaged vehicle. For instance, based on the Y/M/M (and optionally thetrim style) of the depicted damaged vehicle (e.g., the Y/M/M 208), theblock 710 may include accessing stored data, such as stored data in abase object model or a segment model, such as models 120 and 121 of FIG.1, which includes physical principal axis measurements of vehiclesegments of a physical vehicle having the same Y/M/M (and optionally ofthe same trim style) as that of the depicted damaged vehicle, andcomparing the stored physical principal axis measurement of the vehiclesegment with the principal axis measurement of the vehicle segment asdepicted in the source image. In some implementations of the block 710,the method 700 may omit determining the respective depth of fieldindicators of depicted vehicle segments which have respective principalaxes that are only partially visible or not visible at all within thesegmentation map. That is, the method 700 may determine respective depthof field indicators of only the depicted vehicle segments which havefully or entirely visible principal axes within the segmentation map.

Of particular interest are depictions of wheel segments within thesegmentation map. As wheels are essentially round, a depiction orsilhouette of a wheel within an image generally has an elliptical shape.As such, at the block 710, the method 710 may include fitting an ellipseto a wheel segment depicted in the segmentation map, and utilizing thelength of the major axis of the fitted ellipse as the length of theprincipal axis of the depicted wheel segment, e.g., when the fit of theellipse is above a predetermined fit threshold such as 70%, 80%, 90%,etc. When the fit of the ellipse to the depicted wheel silhouette isgreater than the fit threshold, the length of the major axis of thefitted ellipse corresponding to the depicted wheel segment may becompared with the stored length of a diameter of a correspondingphysical wheel to determine the depth of field of the depicted wheelsegment. As such, wheel segments having fully-depicted major axes withinthe source image (even if the entirety of the wheel is not fullydepicted) may be assessed for their respective depths of field.

Upon determining the respective depth of field indicators of theplurality of vehicle segments having fully-depicted principal axeswithin the segmentation map, at the block 710 the method 700 may includeselecting one of the respective depth of field indicators to be thedepth of field indicator corresponding to the segmentation map (andthus, corresponding to the heat map and the target image as well) as awhole. The selection may be made based on one or more various criteria,such as the number of vehicle segments for which respective depth offield indicators where determined, and/or other criteria. For example,when a total number of vehicle segments for which respective depth offield indicators were determined is less than a pre-determined threshold(e.g., two segments, three segments, five segments, etc.), the smallestdepth of field (e.g., the closest to the camera) among the determined,respective depth of field indicators may be selected to be the depth offield indicator corresponding to the source image as a whole. When thetotal number of vehicle segments for which respective depth of fieldindicators were determined is greater than or equal to thepre-determined threshold, the method 700 may include performing aregression analysis on the vehicle segments for which respective depthsof field were determined, and selecting the depth of field indicatorcorresponding to the source image as a whole based on the regressionanalysis. At any rate, the selected depth of field indicatorcorresponding to the source image as a whole may be stored, e.g., in thememories 139 or the data store 215.

At a block 712, the method 700 includes overlaying the segmentation mapand the heat map to form an overlaid image or map of the damagedvehicle. As previously discussed, the segmentation map indicates, foreach pixel, a corresponding vehicle segment represented by the pixel,and the heat map indicates, for each pixel, a corresponding occurrenceand/or types of damage at a corresponding location of the damagedvehicle represented by the pixel. As such, the pixels of the overlaidimage indicate (as represented by various pixels of the overlaid image)the locations of vehicle segments at which damage occurred, andoptionally the one or more types of damaged which occurred at thelocations. Accordingly, the method 700 determines 715 one or moredamaged areas of one or more vehicle segments based on the overlaidimage.

Moreover, at a block 718, the method 700 includes precisely measuring ordetermining the size and/or the location of the detected damaged area.In particular, based on the segmentation map of the damaged vehicle andthe stored measurement of the corresponding physical vehicle, the method700 may determine the units of actual physical length represented byeach pixel of the segmentation map, e.g., based on the principal axiscomparison, or based on a comparison of some other portion, part, edge,dimension, or distance between selected or pre-defined measurementwaypoints included in the depiction of the vehicle and the correspondingphysical measurement of the corresponding portion, part, edge,dimension, or distance between the corresponding waypoints of thecorresponding physical vehicle. For example, at the block 718, themethod 700 may determine the number of pixels which represent a unit ofphysical length of the actual, physical vehicle, such as meters, feet,inches, centimeters, a unit less than centimeter, etc. Subsequently,given the conversion factor of pixels to unit length determined based onthe segmentation map, the method 700 may apply the conversion factor tothe damage indicated by the pixels of the heat map to determine ormeasure the size (e.g., length, width, etc.) of areas of damage on theactual, physical vehicle. Additionally, the method 700 may easilydetermine a precise location of the area of damage on the physicalvehicle, e.g., the distance from a landmark or waypoint of the physicalvehicle, by using the conversion factor and the overlaid segmentationand heat maps. For example, the overlaid image may include pixels thatindicate the damaged area of a vehicle segment as well as pixels whichindicate the relative location of a landmark (such as the edge of thevehicle segment, a corner of another vehicle segment, etc.), and themethod 700 may utilize the conversion factor to precisely measure, inunits of length, the distance of (an edge of) the damaged area from thewaypoint on the physical vehicle.

In some situations, a damaged area may extend across or otherwise appearin multiple images of the damaged vehicle. For example, damage to aright front bumper area may be depicted in both Right Front Corner viewas well as in a Front view of the damaged vehicle, or damage to apassenger door may be depicted in both a Right Side view as well as aRight Side view with a greater degree of zoom. As such, at the block720, the method 700 may include determining whether or not a particulararea of damage is depicted in multiple target images of the damagedvehicle (e.g., based on respective overlaid images of the multipletarget images). When the method 700 determines that a particular area ofdamage is entirely depicted within a single target image (e.g., the “No”leg of block 720), the method 700 may proceed to measure 718 the damagedarea, such as in a manner described above.

On the other hand, when the method 700 determines that a particular areaof damage is depicted in two different target images of the damagedvehicle (e.g., the “Yes” leg of block 720), the method 700 may includestitching together the two target images, the respective segmentationmap images of the two target images, or the respective overlaid mapimages of the two target images so that the damaged area is integrallydepicted within the stitched image (block 722). The two images maydepict adjacent and overlapping portions of the damaged area, and/or thetwo segmentation maps images may depict the damaged area with differentlevels of zoom.

For ease of discussion and not for limitation purposes, the stitchingdiscussion refers to stitching together two overlaid images to form anintegral image. As each of the two overlaid images has been generated bya respective instance of the method 700, the respective depth of fieldof each of the two overlaid images is known, e.g., is stored in thememories 139 or the data store 215. Stitching 722 the two imagestogether may include normalizing the two depths of field into anormalized (e.g., common or same) depth of field for the stitched image,and subsequently adjusting (e.g., resizing, while maintaining therespective aspect ratio) each of the two overlaid map images (ifnecessary) to be sized in accordance with the normalized depth of field.Further, to stitch together 722 the two overlaid images, the method 700may include determining one or more locations (e.g., stitchingwaypoints, features, etc.) of the damaged vehicle that are depicted inboth of the overlaid images. The one or more locations may includelocations of stitching waypoints or features of a segment (or of theentire image) on which at least a part of the depiction of the damagedarea is present, e.g., the edge of the damaged area, a corner of thedamaged area, etc. Additionally or alternatively, the one or morelocations may include locations of stitching waypoints or featuressituated on an area of the vehicle other than the depicted damaged area,such as a depicted roof line, a door handle, or the top of the wheelwhen the damaged area is a rear quarter panel. The method 700 mayutilize any suitable technique to determine the one or more locationswhich are depicted on both of the images, such as convolving,k-dimensional tree filtering, and/or other suitable techniques.

The block 722 may include aligning the determined one or more locationsdepicted in the first image having the normalized depth of field withthe determined one or more locations depicted in the second image havingthe normalized depth of field, and joining or stitching the two imagestogether based on the alignment, thereby forming a single, stitchedimage including features and information from both of the images, andincluding an integral depiction of the damaged area.

Subsequently, the method 700 may assess or analyze the stitched image todetermine additional information pertaining to the damaged area whichwas not available from assessing each of the original imagesindividually. For example, the method 700 may measure the precise sizeand/or location of the damaged area as depicted within the stitchedimage, which may be larger than individually depicted in each of the twoimages. Further, when one of the original images is more zoomed in thanthe other image, additional pixel-related information may be determinedand aggregated with the other image. For example, the more zoomed-inimage has a larger number of pixels per unit length, more detailregarding per-pixel damage presence and/or damage types may be added toor aggregated with that of the less zoomed-in image.

FIG. 8A depicts an example Warping module 800 which may be included inthe image processing system 100, the image processing system 200, and/orin another other image processing system. For example, the Warpingmodule 800 may be included in the warping routine 195 and/or the damagedetailer routine 190 of FIG. 1, or the Warping module 800 may beincluded in the image processing system 200 of FIG. 2. For ease ofdiscussion, though, and not for limitation purposes, the Warping module800 is discussed below with simultaneous reference to FIGS. 1, 2, and3A-3G.

Generally speaking, the Warping module 800 image processes or operateson a source image 802, which may be, for example, a target digitalimage, such as one of the target images 142, the image 202, or the image302, or may be an image generated from the target digital image, such asthe segmentation map image 370 generated from the image 302. The sourceimage 802 depicts various three-dimensional aspects of an objectdepicted within the image 802, such as a vehicle, and the warping module800 image processes the source image 802 to generate a correspondingtwo-dimensional, “flattened” or warped representation 802′ of at least aportion of the object depicted in the source image 802. That is, theWarping module 800 may warp a perspective view of an object into aflattened perspective view of at least a portion of the object. Forexample, the Warping module 800 may transform a source image 802 whichincludes multiple, different depths of field of a depicted object (e.g.,along an edge, axis, or perspective line of the depicted object) into aflattened, essentially two-dimensional representation 802′ of thedepicted object in which the depth of field is uniform (or essentiallyuniform) across at least a planar portion of the depicted object, if notmost or all of the depicted object. Referring to FIG. 3F to illustrate,the Warping module 800 may transform the source image 302 into thewarped source image 302′, and/or may transform the source image 370 intothe warped source image 370′. Additionally, the Warping module 800 mayalso discover, determine, or define the warping transformation 805corresponding to the warping of the input source image 802 into theoutput warped image 802′. Subsequently, the Warping module 800 mayutilize the discovered warping transformation 805 to transform thetarget digital image and/or to transform other images generated from thetarget digital image, such as heat map images.

In an example embodiment, the Warping module 800 includes a set ofcomputer-executable warping instructions 808 that are stored on one ormore tangible, non-transitory memories and executable by one or moreprocessors. For example, when the Warping module 800 is included in theimage processing system 100, the warping instructions 808 may be storedon the one or more memories 139 of the system 100 and may be executableby the one or more processors 138 of the system 100. In otherembodiments, the Warping module 800 may be implemented by using anysuitable, particularly configured combination of firmware and/orhardware in addition to or instead of the computer-executableinstructions 808.

In embodiments, the Warping module 800 may be executable to perform atleast a portion of an example method 820 for warping a source imagewhich depicts three-dimensional features, characteristics, or aspects ofan object, a block flow diagram of which is depicted in FIG. 8B. Forease of discussion, and not for limitation purposes, the method 820 isdiscussed below with simultaneous reference to the systems, modules, andother components shown in FIGS. 1, 2, 3A-3G, and 8A, although the method820 may be implemented in other systems and/or by other modules andcomponents. Additionally, although FIG. 8B depicts the method 820 asincluding the blocks 822-830, in other embodiments the method 820 mayinclude fewer, additional, or alternate steps.

As shown in FIG. 8B, at a block 822 the method 820 includes determiningthe respective locations, within a source image such as the source image802 of FIG. 8A, of a plurality of warping waypoints corresponding to anobject depicted in the source image (such as a vehicle). As previouslydiscussed, the source image on which the method 820 operates may be adigital image captured by a camera or may be an image generated from thecaptured digital image, such as segmentation map or segmented image.Generally speaking, and as previously discussed, the source image maydepict various three-dimensional aspects of an object or vehicledepicted therein, such as multiple, different perspective lines alongthe object corresponding to different vanishing points of the sourceimage. That is, a source image may depict a perspective view of theobject or vehicle. For example, referring to the digital image 302 ofFIG. 3F as an example source image, a first perspective line associatedwith the vehicle 305 extends from the right front corner of the vehicletowards the right rear corner of the vehicle 305, and a secondperspective line associated with the vehicle 305 extends from the rightfront corner of the vehicle towards the left front corner of the vehicle305, e.g., in a direction different than the direction of the firstperspective line, and towards a different vanishing point than thevanishing point of the first perspective line. The respective depth offield corresponding to the first perspective line varies along itsrespective direction (e.g., a first three-dimensional aspect of thevehicle 305), and the respective depth of field corresponding to thesecond perspective line varies along its respective direction (e.g., asecond three-dimensional aspect of the vehicle 305). Of course, otherthree-dimensional aspects of the vehicle 305 may be depicted within thesource image 302.

In an embodiment, the determining 822 of the respective locations of theplurality of warping waypoints corresponding to the depicted object mayinclude determining the respective locations of a pre-defined set ofwarping waypoints. Different sets of pre-defined warping waypoints maybe determined a priori for different segments, and indications of thesets of pre-defined warping waypoints and their corresponding segmentsmay be stored in database 108, for example. For instance, when thedepicted object is a vehicle, the pre-defined warping waypoints mayinclude the four corners of a depicted side panel or a door, and whenthe depicted object is a building, the pre-defined warping waypoints mayinclude the corners of a depicted front or side elevation, or of a dooror window depicted in the front or side elevation. Referring to FIG. 3Gto illustrate, an example source image 390 of a target vehicle is aperspective view of a segment map of the target vehicle, where thesegment map may have been generated, by using any of the techniquesdescribed elsewhere herein, from another, two-dimensional digital imagecaptured by a camera. A target segment of the source image 390 forwarping may be an entire planar view 392 a of the target vehicle (inthis case, the passenger or right-side view 392 a of the target vehicle,which is depicted separately in FIG. 3G from the depiction of sourceimage 390 for clarity of illustration), or may be a portion 392 b of theplanar view 392 a, e.g., as illustrated in FIG. 3G, a portion or segment392 b representing the passenger door of the vehicle. As an example, aset of pre-defined warping waypoints for the segment 392 a and/or forthe segment 392 b may include the four corners of the passenger door. Assuch, in this embodiment, the block 822 may include determining therespective locations of four corners the passenger door as depictedwithin the source image 390, where the pre-determined set of warpingwaypoints are included in the target segment 392 for warping. For somesegments, though, the pre-defined set of warping waypoints may includeone or more warping waypoints which are not included in the targetsegment. For example, if a target segment is a frame of a vehicledisposed between the front and the rear passenger side doors, the set ofpre-defined warping waypoints may include waypoints located on either orboth of the passenger side doors, or if the target segment is theright-side planar view, a pre-defined warping waypoint may include awaypoint located on the back-side planar view of the vehicle.

In an embodiment (not shown in FIG. 8B), the determining 822 of therespective locations of a plurality of warping waypoints mayadditionally or alternatively include determining or discovering (e.g.,in-line with the execution of the method 820) respective locations of aplurality of optimal or optimized waypoints for warping the depiction ofthe target object (or segment thereof) within the source image 802. Thatis, the method 820 may include discovering a set of warping waypointswhich particularly optimize the warping of the source image 802 (orsegment thereof) and the respective locations of the waypoints withinthe source image 802. Said another way, determining 822 the respectivelocations of a plurality of warping waypoints for the source image 802may include discovering an optimized set of warping waypoints that isunique to the source image 802. Similar to the pre-defined sets ofwarping waypoints, one or more discovered warping waypoints may beincluded in a target segment that is to be warped, and/or one or morediscovered warping waypoints may be included in a different segment.Further, the discovery of the optimized set of waypoints may be based onthe segment of the source image 802 that is targeted for warping (whichmay be the entire image 802 or a portion thereof) and a standard view ofa physical object corresponding to object depicted in the image 802.

Generally speaking, a “standard view” of a physical object, such as avehicle, may be a planar view of the object that has an essentiallyuniform depth of field with respect to a camera that captured or couldhave captured the planar view. As such, a standard view of a physicalobject may be a standard right side view, a standard left side view, astandard front view, or a standard back view of the physical object. Asutilized herein, a “standard” view is a view that is generated from aplurality of different images of the same type of planar view (e.g.,right side, left side, front, back) of similar physical objects, forexample, vehicles of a same make, model, and year of manufacture. Forexample, a standard right side planar view of a vehicle of a particularmake, model, and year of manufacture may be generated from a pluralityof different images of right side planar views of different vehicles ofthe particular make, model, and year of manufacture, where the pluralityof different images may include images of undamaged vehicles and mayinclude images of damaged vehicles. As such, the “standard” view of thevehicle may be considered to be a standardization, combination,amalgamation, or averaged representation of the same type of planar viewof multiple vehicles of the particular make, model, and year ofmanufacture. To illustrate, FIG. 3G depicts an example standard rightside view 395 of a vehicle of the same make, model, and year ofmanufacture as the subject vehicle depicted in source image 390, wherethe standard right side view 395 has been generated from a plurality ofimages of right side view of different vehicles of the same make, model,and year of manufacture as the subject vehicle depicted in the sourceimage 390. Indeed, in some embodiments (not shown), the method 820 mayinclude generating one or more standard views of a vehicle of aparticular make, model, and year of manufacture, and storing thegenerated standard views, e.g., in one or more data storage entities108.

At any rate, in this embodiment, discovering the plurality of optimizedwarping waypoints based on the segment of the target object and thestandard view of a corresponding physical object may include determininga particular planar view included in the source image 802 (e.g., front,back, right, left), where the particular planar view typically hasmultiple, different depths of field as the source image is a perspectiveview of the object. Additionally, discovering the optimized warpingwaypoints may further include mapping the particular planar viewdepicted within the source image 802 to a standard particular planarview of a physical object corresponding to the object depicted in thesource image 802. Additionally, this embodiment of the block 822 mayinclude applying a machine-learning optimization technique to themapping to discover the respective locations of the optimized set ofwarping waypoints. The machine-learning technique may be a reinforcementlearning technique, a Monte Carlo simulation, or another type ofoptimizing machine-learning technique. For example, using themachine-learning technique, multiple warpings of the source image may beiteratively performed using different sets of candidate warpingwaypoints, and each resultant warped image may be evaluated against thestandard view based on the mapping. In an example implementation, themachine-learning technique may discover the set of warping waypointswhich minimize the space or distance between the source image 802 andthe standard view, e.g., by maximizing a ratio between the intersectionof the source image 802 and the standard view and a union of the sourceimage 802 and the standard view. In some embodiments, themachine-learning technique may also determine a number of warpingwaypoints included in the optimized set, e.g., the optimized number ofwarping waypoints for the source image 802. Typically, the number ofwarping waypoints is at least three waypoints, although an optimumnumber of warping waypoints may be discovered to be three, four, five,or more waypoints. The machine-learning technique may be bounded, suchas by a threshold level, a maximum number of attempts, etc. In someembodiments, multiple machine-learning techniques may be utilized.

At a block 825, the method 820 includes reshaping a minimum area polygonthat corresponds to the determined warping waypoints and that covers thedepiction of the planar portion of the object (e.g., of the particularplanar view) within the source image 802. The reshaping 825 may be basedon physical measurements and an aspect ratio of a corresponding planarportion of a physical object corresponding to the object depicted in theimage. Referring to FIG. 3F to illustrate, the block 825 may includereshaping a minimum area polygon corresponding to the segment 380 of thevehicle 305 indicative of the passenger door and its four corners 385 a,385 b, 385 c, 385 d (which were previously determined at the block 822to be the respective locations of the set of warping waypoints) into aminimum area polygon which is accordance with the aspect ratio of aphysical passenger door of a corresponding physical vehicle. In someembodiments, at the block 825, the method 820 may include determining orgenerating the minimum area polygon that covers the segment 380 and thatincludes the locations of the four corners 385 a-385 d, e.g., by usingany suitable method, prior to reshaping.

Additionally, based on the Y/M/M 308 of the vehicle 305, the method 820may determine the aspect ratio of a physical passenger door of aphysical vehicle having the same Y/M/M 308 as the vehicle 305, e.g., byaccessing stored data indicative of the physical measurements andoptionally of the corresponding aspect ratio of the passenger door of aphysical vehicle having the same Y/M/M 308 as the vehicle 305 which maybe stored, for instance, in database 108. Based on the aspect ratio ofthe corresponding physical vehicle and based on the determined locationsof the four corners 385 a-385 d within the source image 370, the method820 may reshape the minimum area polygon covering the segment 380 of thesource image 370 to have the same aspect ratio as that of the passengerdoor of the corresponding physical vehicle.

Generally speaking, the warping of the minimum area polygon to have theaspect ratio of a corresponding segment or portion of a correspondingphysical vehicle may specify or indicate the warping transformation ofthe minimum area polygon. Accordingly, at the block 828, the method 820may include determining (and optionally storing or saving) the warpingtransformation, e.g., as warping transformation 805. Generally speaking,an application of the warping transformation to the initial minimum areapolygon results in the warped minimum area polygon having the sameaspect ratio as a corresponding physical segment of a physical object orvehicle corresponding to the object or vehicle depicted in the sourceimage.

At a block 830, the method 820 may include applying the warpingtransformation to the source image, e.g. to the source image in itsentirety or to each segment included in the source image. As such, atthe block 830, the method 820 thereby generates a warped source imagewhich has an (essentially) uniform depth of field across the planarportion of the depicted object, if not most or all of the depictedobject. For example, referring to FIG. 3F, upon determining the warpingtransformation corresponding to the warped minimum area polygon of thesegment 380 indicative of the passenger door, the warping transformationmay be applied to the source image 370 to thereby generate the warpedsource image 370′ in which the depth of field is uniform across thesegment 380′ indicative of the passenger door, and in some cases, isuniform across more than one or all of the segments of the vehicle 305depicted within the warped source image 370′.

Further, in some embodiments, the method 820 may include applying thewarping transformation to other images generated from or which areotherwise based on the source image 370 (not shown in FIG. 8B), forexample, that are of the same perspective of the camera angle and aresized to the size of the source image. For example, when the sourceimage is a segmentation image of a vehicle on which the method 820operates to generate a corresponding warped segmentation image and awarping transformation, the method 820 may further include applying thewarping transformation to the target digital image from which thesegmentation image was generated and/or to a heat map image generatedfrom the target digital image corresponding to the segmentation image.In some situations, the method 820 may include applying the warpingtransformation to an overlaid image, such as to an overlay of the targetdigital image masked by the corresponding heat map, or to some othercombination of overlaid images of the object which are based on thetarget digital image of the object.

Beneficially, the warped source image 370′ is an image which may beutilized by image processing systems to precisely measure actualcharacteristics of the damaged vehicle 305, at least due to therelatively uniform depth of field across the damaged vehicle depicted inthe image 370′. Moreover, also at least due to the uniform depth offield of the warped image 370′, the warped image 370′ may easily becombined or stitched with other images to form a stitched image based onwhich aggregate damage information corresponding to damaged areas thatextend across multiple images may be determined.

FIG. 9A depicts an example Stitching module 900 which may be included inthe image processing system 100, the image processing system 200, and/orin another other image processing system. For example, the Stitchingmodule 900 may be included in the stitching routine 197 and/or thedamage detailer routine 190 of FIG. 1, or the Stitching module 900 maybe included in the image processing system 200 of FIG. 2. For ease ofdiscussion, though, and not for limitation purposes, the Stitchingmodule 900 is discussed below with simultaneous reference to FIGS. 1, 2,and 3A-3G.

Generally speaking, the Stitching module 900 image processes or operateson two source images 902 a, 902 b depicting a same object, such as avehicle, to generate a single, integral stitched image 905 whichincludes features and information from each of the source images 902 a,902 b. The two source images 902 a, 902 b may be, for example, twotarget digital images, such as a first and a second target image 142 ofFIG. 1, two segmentation map images respectively generated from thefirst and the second target images 142, or a segment from the firsttarget image and a segment from the second target image. Generallyspeaking, each of the source images 902 a, 902 b has a respectiveuniform depth of field, and thus may be a side view image of the objector vehicle, or may be flattened perspective view image of the object orvehicle. As such, in some cases, at least one of the source images 902a, 902 b may have been generated by the Warping module 800 of FIG. 8.Each of the source images 902 a, 902 b depicts at least a respectiveportion of a same particular portion or segment of the object orvehicle. For example, the source image 902 a may be segmentation mapimage of a warped, Right Rear Corner view of a vehicle, and thus maydepict at least a portion of a segment indicative of the right rearquarter panel of the vehicle, and the source image 902 b may be asegmentation image of a Right Side view of the vehicle and thus maydepict an entirety of the segment indicative of the vehicle's right rearquarter panel.

In an example embodiment, the Stitching module 900 includes a set ofcomputer-executable stitching instructions 908 that are stored on one ormore tangible, non-transitory memories and executable by one or moreprocessors. For example, when Stitching module 900 is included in theimage processing system 100, the stitching instructions 908 may bestored on the one or more memories 139 of the system 100 and may beexecutable by the one or more processors 138 of the system 100. In otherembodiments, the Stitching module 900 may be implemented by using anysuitable, particularly configured combination of firmware and/orhardware in addition to or instead of the computer-executableinstructions 908.

In embodiments, the Stitching module 900 may be executable to perform atleast a portion of an example method 920 for stitching together twosource images of an object, a block flow diagram of which is depicted inFIG. 9B. For ease of discussion, and not for limitation purposes, themethod 920 is discussed below with simultaneous reference to thesystems, modules, components, and methods shown in FIGS. 1, 2, 3A-3G,8A-8B, and 9A, although the method 920 may be implemented in othersystems and/or by other modules and components, and in conjunction withother methods. Additionally, although FIG. 9B depicts the method 920 asincluding the blocks 922-940, in other embodiments the method 920 mayinclude fewer, additional, or alternate steps.

As shown in FIG. 9B, at a block 922 the method 920 includes obtainingtwo source images of an object, such as a vehicle. Each of the twosource images depicts at least a portion of a same area, segment, orcomponent of the object. The two source images may both be digitaltarget images, such as digital target images 302 and 320, or the twosource images may both be segmentation map images, such as segmentationimages 370 and 372. The images may be obtained from a data store, suchas data store 109 or the memories 139, or the images may be obtained viaa network or communication interface.

At a block 925, the method 920 includes determining if either of theobtained source images is a perspective view image of the object. If so(e.g., the “Yes” leg of block 925), then the method 920 includes warping928 each of the perspective view images into a flattened perspectiveview image having a uniform depth of field. For example, the method 920may be utilized to warp each of the perspective view images. After anyperspective view images have been warped, or when the two source imagesdo not include any perspective views (e.g., the “No” leg of block 925),the method 920 includes determining 930 whether or not the two images(at this point both of which have respective relatively uniform depthsof field) have equivalent depths of field. If not (e.g., the “No” leg ofblock 930), then the method 920 includes normalizing 932 or otherwisetruing up the two different depths of field. Normalizing 932 the depthsof field into a common or equivalent depth of field may include resizingone or both of the source images, in some situations.

Upon both images having a same normalized or equivalent depth of field(e.g., as denoted by the exit arrow of block 932 or the “Yes” leg ofblock 930), the method 920 includes determining 935 one or more commonstitching waypoints or features which are depicted in both sourceimages. Stitching waypoints or features may include, for example, acorner, an edge, or other feature of the area, segment, or component ofthe object which is at least partially depicted in each of the twosource images. Additionally or alternatively, stitching waypoints orfeatures may include waypoints or features of other areas, segments, orcomponents of the object. For example, when the commonly depictedsegment in both images is a damaged right front bumper, the waypointsmay include the right headlight assembly, the right front wheel, etc.The one or more common stitching waypoints/features may be determinedusing any suitable technique, such as convolution, utilizingk-dimensional trees, etc.

At a block 938, the method 920 includes aligning the two images havingthe normalized depths of field based on the determined common stitchingwaypoints, and at a block 940, the method 920 includes stitchingtogether the aligned images to form a single, stitched image (e.g., thestitched image 905) depicting the area, segment, or component of theobject which was at least partially depicted in each of the obtainedsource images. For example, when each source image depicts a differentportion of a same segment of the object which partially overlap, thestitched image depicts an aggregate depiction of a larger portion (andin some cases, the entirety) of the segment in a single image. Inanother example, when one of the source images depicts a smaller portionof the segment than the other sourced image, but with a greater level ofzoom than the other source image, the stitched image depicts theadditional detail provided by the greater level of zoom into the largerdepiction of the segment.

The stitching technique of method 920 is particularly useful formanipulating images of a segment which wraps around or otherwise extendsover several views of an object, such as the front bumper assembly or afront headlight assembly of an automobile, into a single view via whichcharacteristics of a damaged area of the segment whose depiction extendsacross the multiple views may be accurately determined. For example,precise measurements of the entire size and location of the damaged areamay be determined from the stitched image, and/or particular types ofdamage to the segment may be identified from the stitched image.

As previously discussed, each of the set of target images of an objectmay be operated on by the image processing system 100, the imageprocessing system 200, and/or by one or more of the disclosed imageprocessing techniques to detect changes to the depicted object, and thedetected change information (e.g., type(s) of change(s), preciselocation(s) of change on the object, precise size(s) of changes, etc.)may be consolidated and provided to a user interface for displaythereon, stored in a file, and/or transmitted to another computingdevice. FIGS. 10A-10C depict example outputs and display screens (whichmay be presented on user interfaces, for example), each of whichprovides damage information determined from image processing a set oftarget images of a damaged vehicle, e.g., via the image processingsystem 100, the image processing system 200, and/or by using one or moreof the disclosed image processing techniques for detecting changes indepicted objects.

FIG. 10A illustrates an example implementation of aggregating andconsolidating damage or change information detected from imageprocessing multiple images of a damaged vehicle. In FIG. 10A, originaltarget image 950 of a damaged vehicle 955 has been overlaid with acorresponding heat map 958. Damaged areas depicted in the image 950 havebeen typed as being a “dent” on the right front bumper (“site_0”) and a“scratch” on the right front bumper (“site_1”), and have beenrespectively measured based on a corresponding segmentation map 960.Similarly, original target image 965 of the damaged vehicle 955 has beenoverlaid with a corresponding heat map 968. The damaged areas depictedin image 955 have been identified as a “scratch” on the right frontbumper (“site_0”) and a “dent” on the grille (“site_1”), and have beenrespectively measured based on a corresponding segmentation map 970. Asdamage to the right front bumper has been detected in both image 950 andimage 965, in the consolidated information, the front bumper need onlybe indicated once as a damage site (and not three times, as denoted bythe slanted lines) in conjunction with its corresponding (aggregate andconsolidated) types of damages (e.g., “dent” and “scratch”) along withthe indication of the grille and its corresponding type of damage (e.g.,“dent”).

FIG. 10B depicts an example display screen 975 on which detected changeinformation of a vehicle may be presented to a user. The display screen975 presents four segmented views of the vehicle, e.g., a front view 978a, a rear view 978 b, a left side view 978 c, and right side view 978 d,where each segment of the vehicle is depicted a different color, and thecolor key is consistent across the four views 978 a-978 d. Based on thepixels of the heat map images and the segmentation map images generatedbased on the initial set of target images of the vehicle, the detectedareas of damage are indicated in the views 978 a-978 d by the colorblack. A user may zoom in on any of the views to obtain a more detaileddepiction of a damaged area (if any), and/or a user may select a damagedarea (e.g., by clicking), upon which detailed information about thedamaged area (e.g., precise size, segment(s) affected by the damage,precise location, damage type(s), etc.) may be displayed (e.g., in apop-up window, or other suitable representation). In some embodiments,by selecting an indicated area of damage on one of the views 978 a-978d, the display screen 975 may present, on the user interface, theoriginal target image(s) which include the selected area of damage. Ofcourse, such damage information associated with display view 975 may bestored in a file and processed by another computing device or routine,e.g., to determine replacement parts to order, to determine insuranceestimates, to estimate repair costs, etc.

FIG. 10C depicts a display screen 990 on which detected changeinformation of a vehicle may be presented to a user. The display screenpresents a “butterflied” depiction of the vehicle in which five views ofthe vehicle, e.g., a front view 992 a, a rear view 992 b, a left sideview 992 c, a right side view 992 d, and a bird's eye view 992 e) aredepicted by using outlines of segments and parts of the vehicle whichare visible in each view. Detected areas of damage 995 (in this case,only a single detected area of damage) are depicted on the displayscreen 990 in a contrasting color. Similar to the display screen 975, auser may zoom in on any of the views to obtain a more detailed depictionof a damaged area (if any), and/or a user may select the damaged area(e.g., by clicking), upon which detailed information about the damagedarea (e.g., precise size, segment(s) affected by the damage, preciselocation, damage type(s), etc.) may be displayed (e.g., in a pop-upwindow, or other suitable representation). In some embodiments, byselecting an indicated area of damage on one of the views 992 a-992 e,the display screen 990 may present, on the user interface, the originaltarget image(s) which include the selected area of damage. Of course,such damage information associated with display view 990 may be storedin a file and processed by another computing device or routine, e.g., todetermine replacement parts to order, to determine insurance estimates,to estimate repair costs, etc.

Of course, FIGS. 10A-10C are exemplary outputs and display screens onwhich the change information detected by the disclosed image processingtechniques may be presented. Other display screens and representationsmay be utilized.

The following additional considerations apply to the foregoingdiscussion. Throughout this specification, plural instances mayimplement components, operations, or structures described as a singleinstance. Although individual operations of one or more routines ormethods are illustrated and described as separate operations, one ormore of the individual operations may be performed concurrently, andnothing requires that the operations be performed in the orderillustrated. Structures and functionality presented as separatecomponents in example configurations may be implemented as a combinedstructure or component. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents. These and other variations, modifications, additions, andimprovements fall within the scope of the subject matter of the presentdisclosure.

Additionally, certain embodiments are described herein as includinglogic or a number of components, modules, or mechanisms or units. Any ofthese modules, units, components, etc. may constitute either softwaremodules (e.g., code stored on a non-transitory, tangible,machine-readable medium) or hardware modules. A hardware module istangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client or server computersystem, cloud computing system, etc.) or one or more hardware modules ofa computer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa hardware module that operates to perform certain operations asdescribed herein.

A hardware module may comprise dedicated circuitry or logic that ispermanently configured (e.g., as a special-purpose processor, such as afield programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC)) to perform certain operations. A hardwaremodule may also include programmable logic or circuitry (e.g., asencompassed within a general-purpose processor or other programmableprocessor) that is temporarily configured by software to perform certainoperations. It will be appreciated that the decision to implement ahardware module in dedicated and permanently configured circuitry or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the hardware terms used herein should be understood toencompass tangible entities, be that entities that are physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware and software modules or routines can provide information to,and receive information from, other hardware and/or software modules androutines. Accordingly, the described hardware modules may be regarded asbeing communicatively coupled. Where multiple of such hardware orsoftware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits, lines andbuses) that connect the hardware or software modules. In embodiments inwhich multiple hardware modules or software are configured orinstantiated at different times, communications between such hardware orsoftware modules may be achieved, for example, through the storage andretrieval of information in memory structures to which the multiplehardware or software modules have access. For example, one hardware orsoftware module may perform an operation and store the output of thatoperation in a memory device to which it is communicatively coupled. Afurther hardware or software module may then, at a later time, accessthe memory device to retrieve and process the stored output. Hardwareand software modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, includeprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method may be performed by one or processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a university complex, anoffice environment, or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

Some portions of this specification are presented in terms of algorithmsor symbolic representations of operations on data stored as bits orbinary digital signals within a machine memory (e.g., a computermemory). These algorithms or symbolic representations are examples oftechniques used by those of ordinary skill in the data processing artsto convey the substance of their work to others skilled in the art. Asused herein, an “application,” an “algorithm” or a “routine” is aself-consistent sequence of operations or similar processing leading toa desired result. In this context, applications, algorithms, routinesand operations involve physical manipulation of physical quantities.Typically, but not necessarily, such quantities may take the form ofelectrical, magnetic, or optical signals capable of being stored,accessed, transferred, combined, compared, or otherwise manipulated by amachine. It is convenient at times, principally for reasons of commonusage, to refer to such signals using words such as “data,” “content,”“bits,” “values,” “elements,” “symbols,” “characters,” “terms,”“numbers,” “numerals,” or the like. These words, however, are merelyconvenient labels and are to be associated with appropriate physicalquantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of “a” or “an” is employed to describe elements andcomponents of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription should be read to include one or at least one and thesingular also includes the plural unless it is obvious that it is meantotherwise.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs may beused for implementing an image processing application and system forconfiguring and executing the change detection techniques disclosedherein. Thus, while particular embodiments and applications have beenillustrated and described herein, it is to be understood that thedisclosed embodiments are not limited to the precise construction andcomponents disclosed herein. Various modifications, changes andvariations, which will be apparent to those skilled in the art, may bemade in the arrangement, operation and details of the methods andstructure disclosed herein without departing from the spirit and scopedefined in the claims.

What is claimed:
 1. An image processing tool, comprising: a computerstorage unit for storing one or more electronic images of one or morevehicles; an electronic image editing tool including; a user displaythat displays the one or more electronic images of the one or morevehicles; a user input device that enables a user to select one or moreareas of an electronic image of a vehicle currently being displayed onthe user display by specifying a set of pixels associated with theselected area within the displayed electronic image; and a markingengine that marks each of the specified set of pixels associated withthe selected area with a vehicle characteristic; and an image storagedevice that stores an edited electronic image corresponding to thedisplayed electronic image, the edited electronic image including imagepixel data of the displayed electronic image and an indication of thevehicle characteristic for each of the marked pixels in the selectedarea of the displayed electronic image.
 2. The image processing tool ofclaim 1, wherein the vehicle characteristic indicates damage to thevehicle.
 3. The image processing system of claim 1, wherein the vehiclecharacteristic indicates a type of damage to the vehicle at the selectedarea.
 4. The image processing system of claim 1, wherein the vehiclecharacteristic indicates a segment of the vehicle depicted in thedisplayed electronic image.
 5. The image processing system of claim 1,wherein the electronic image editing tool includes a selection modulethat enables a user to specify a particular vehicle characteristic foreach of the set of pixels in the selected area.
 6. The image processingsystem of claim 1, wherein the selection module enables a user tospecify a particular vehicle characteristic as one of a predeterminedset of vehicle characteristics.
 7. The image processing system of claim6, wherein the selection module displays the predetermined set ofvehicle characteristics on the user display and enables a user to selectone of the displayed predetermined set of vehicle characteristics as thevehicle characteristic for the set of pixels of the selected area viathe user input device.
 8. The image processing system of claim 7,wherein the selection module displays the predetermined set of vehiclecharacteristics on the user display via a drop down menu on the displayscreen.
 9. The image processing system of claim 6, wherein thepredetermined set of vehicle characteristics includes different types ofdamage to a vehicle.
 10. The image processing system of claim 9, whereinthe different types of damage to a vehicle includes one or more of adent, a scratch, a tear, a misalignment, a missing piece, a crack, acrease, a bend, a buckle, a twist, a ding or a hole.
 11. The imageprocessing system of claim 6, wherein the predetermined set of vehiclecharacteristics includes different types of vehicle segments.
 12. Theimage processing system of claim 11, wherein the different types ofvehicle segments include one or more of a hood, a side panel, a bumper,a wheel, a rear side and a front side.
 13. The image processing systemof claim 1, further including a model training engine that trains aregression model using multiple ones of the edited electronic images.14. The image processing system of claim 1, further including anestimate unit that executes on a computer processor to estimate a set ofpixels within the displayed electronic image associated with the vehiclecharacteristic, and that displays an indication of the estimated set ofpixels within the displayed image, and wherein the user input toolenables a user to modify or accept the estimated set of pixels to beassociated with the vehicle characteristic.
 15. A method of processingan image, comprising: storing a plurality of electronic images of one ormore vehicles in a database; displaying, via a user display device, oneor more of the plurality of electronic images of the one or morevehicles; enabling a user to select an area within a displayed one ofthe electronic images currently being displayed on the user display byusing a user input device to specify a set of pixels associated with aselected area within the displayed one of the electronic images;marking, via a computer processor, the specified set of pixelsassociated with the selected area with a vehicle characteristic; andstoring, via a computer processing device, an edited electronic imagecorresponding to the edited displayed electronic image, the editedelectronic image including image pixel data of the displayed electronicimage and an indication of the vehicle characteristic of each of themarked pixels in the selected area of the displayed electronic image.16. An image processing tool, comprising: a computer storage unit forstoring a plurality of electronic images of a multiplicity of differentvehicles, each of the electronic images depicting a vehicle havingdamage thereon; an electronic image editing tool including; a userdisplay that displays at least one of the plurality of electronic imagesto a user; a user input device that enables a user to select one or moreareas within an electronic image of a vehicle currently being displayedon the user display by specifying a set of pixels associated with theselected area within the displayed one of the electronic images; and amarking engine, that executes on a computer processor, to mark each ofthe specified set of pixels associated with the selected area with avehicle characteristic; an image storage device that stores editedelectronic images corresponding to the edited displayed electronicimages, the edited electronic images including image pixel data of thedisplayed electronic image and an indication of the vehiclecharacteristic of each of the marked pixels in the selected areas of thedisplayed electronic images; and a model generation routine, forexecution on a computer processor, that creates a regression model froma plurality of edited electronic images stored in the image storagedevice, the regression model trained to detect the vehiclecharacteristic in a new vehicle image on a pixel by pixel basis.
 17. Theimage processing tool of claim 16, wherein the vehicle characteristicindicates damage to the vehicle, or a particular type of damage to avehicle or a particular segment of a vehicle.
 18. The image processingsystem of claim 16, further including an estimate unit that executes ona computer processor to estimate a set of pixels within a displayedelectronic image associated with the vehicle characteristic, and thatdisplays an indication of the estimated set of pixels within thedisplayed image having the vehicle characteristic, and wherein the userinput tool enables a user to modify or accept the estimated set ofpixels to be associated with the vehicle characteristic.
 19. The imageprocessing system of claim 16, wherein each of the plurality ofelectronic images further includes information associated therewithdefining a vehicle view or a zoom level associated with the electronicimage, and wherein the model generation routine creates the regressionmodel using the information defining a vehicle view or a zoom level togenerate the regression model.
 20. The image processing system of claim16, wherein the regression model comprises a convolutional neuralnetwork model.